KR102417257B1 - Apparatus and method for filling electronic document based on eye tracking and speech recognition - Google Patents

Abstract

An apparatus and method for creating an electronic document based on eye tracking and voice recognition are disclosed, and the method for creating an electronic document based on eye tracking and voice recognition according to an embodiment of the present application includes the steps of displaying an electronic document including a plurality of entry items , obtaining the user's gaze information for the displayed electronic document; and writing a field value corresponding to the identified entry item based on the received voice input.

Description

Apparatus and method for creating electronic documents based on eye tracking and voice recognition

The present application relates to an apparatus and method for creating an electronic document based on eye tracking and voice recognition.

In general, various types of documents are generated in the work area, and if these documents were conventionally written in printed form, electronic documents and electronic signature services that generate various documents in electronic form have recently been introduced and utilized. , is showing a rapid growth trend in line with the government's paperless policy.

In addition, with the popularization of smart devices, user experiences using voice-based services have been continuously accumulated. Currently, most of the voice-based services provide functions related to an individual's daily life, and it is expected to expand to the business area of companies in the future.

Accordingly, the need for developing a technology and system for generating an electronic document based on a voice input or the like or a technology and system capable of controlling and managing an electronic document based on a voice input is increasing.

However, the technology for creating electronic documents through voice so far has turned the entire voice into text (transcription, transcription), such as creating subtitles for content (movies, videos uploaded online, etc.) ) was concentrated only on

On the other hand, the most sensitive and quick-response of the body's five senses is the human's most important information acquisition route. In this regard, Eye Tracking (Gaze Tracking) technology detects where the user's gaze is directed. In addition, VR, AR, and integration with various IT devices are actively being carried out, and use in various fields such as next-generation games, analysis of driver's gaze information, financial payment based on iris information recognition, and viewing age restriction function. Although the trend is evaluated to be highly likely, the development and introduction of techniques to assist the creation process of the displayed electronic documents by grafting the technology of tracking the user's gaze are still insufficient.

The technology that is the background of the present application is disclosed in Korean Patent Publication No. 10-1922589.

The present application aims to solve the problems of the prior art described above, and to provide an electronic document creation apparatus and method based on eye tracking and voice recognition for creating an electronic document based on a voice input for a entry item that a user gazes at. The purpose.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the method for creating an electronic document based on eye tracking and voice recognition according to an embodiment of the present application includes the steps of displaying an electronic document including a plurality of entry items, the displayed electronic document Acquiring gaze information of a user for , identifying an entry item that the user gazes on based on the gaze information, receiving the user's voice input for the identified entry item, and the received voice and filling in a field value corresponding to the identified entry item based on the input.

In addition, in the step of identifying the entry item, if it is determined that the user gazed at an area corresponding to a predetermined entry item in the displayed electronic document for a preset threshold time or longer, receiving the voice input associated with the entry item You can switch to the input mode that is possible.

In addition, in the method for creating an electronic document based on eye tracking and voice recognition according to an embodiment of the present application, after the step of writing the field value, the field value based on preset writing order information for the plurality of entry items and displaying the entry items corresponding to the next order of the written entry items to be highlighted.

In addition, the writing order information may be a predefined writing order for a predetermined essential writing item among a plurality of writing items included in the displayed electronic document.

In addition, the displaying of the electronic document may include displaying an entry item corresponding to the first creation item to be highlighted based on the writing order information.

In addition, in the method for creating an electronic document based on eye tracking and voice recognition according to an embodiment of the present application, after the step of identifying the entry item, the voice input is not received within a preset waiting time after switching to the input mode Otherwise, the method may include displaying guide information for the corresponding entry item.

On the other hand, the electronic document creation apparatus based on eye tracking and voice recognition according to an embodiment of the present application obtains a display unit for displaying an electronic document including a plurality of entry items, and information about a user's gaze on the displayed electronic document, Receives a gaze analysis unit for identifying the entry item that the user gazes at based on the gaze information and the user's voice input for the identified entry item, and adds the identified entry item based on the received voice input It may include a voice processing unit for writing a corresponding field value.

In addition, when it is determined that the user gazes at an area corresponding to a predetermined entry item in the displayed electronic document for more than a preset threshold time, the gaze analyzer is configured to receive the voice input associated with the entry item. mode can be switched.

Also, the display unit may display an entry item corresponding to a next order of the entry item in which the field value is written to be highlighted based on preset writing order information for the plurality of entry items.

In addition, the display unit may display a writing item corresponding to the first writing item to be emphasized based on the writing order information.

Also, if the voice input is not received within a preset waiting time after switching to the input mode, the display unit may display guide information for a corresponding entry item.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, it is possible to provide an electronic document creation apparatus and method based on eye tracking and voice recognition that allows a user to create an electronic document based on a voice input for an entry item staring at.

According to the above-described problem solving means of the present application, the electronic document is automatically completed through repetition of the process of the user staring at the entry item included in the displayed electronic document and uttering the content corresponding to the gazed entry item, so that the user can conveniently create electronic documents without additional input means or complicated operations.

According to the above-described problem solving means of the present application, the entry items to be filled in by the user to complete the electronic document are highlighted, and the contents reflected in the electronic document are displayed by the user while the user gazes at the highlighted entry items to create the electronic document. can make you familiar with it.

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a conceptual diagram for explaining the overall operation and function of an electronic document creation apparatus based on eye tracking and voice recognition according to an embodiment of the present application.
2 is a schematic configuration diagram of an electronic document creation apparatus based on eye tracking and voice recognition according to an embodiment of the present application.
3A to 3C illustrate a process of identifying an entry item that the user gazes at based on the user's gaze information with respect to a predetermined electronic document, and entering a field value for the identified entry item based on the user's voice input It is a drawing for
4 is a diagram for explaining guide information for entry items.
FIG. 5 is a diagram for explaining that a preset threshold time is individually set for each entry item based on the amount of content information associated with the entry item; FIG.
6 is an operation flowchart of a method for creating an electronic document based on eye tracking and voice recognition according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily carry out. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including cases where

Throughout this specification, when a member is positioned “on”, “on”, “on”, “on”, “under”, “under”, or “under” another member, this means that a member is positioned on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

The present application relates to an apparatus and method for creating an electronic document based on eye tracking and voice recognition.

1 is a conceptual diagram for explaining the overall operation and function of an electronic document creation apparatus based on eye tracking and voice recognition according to an embodiment of the present application.

Referring to FIG. 1 , an electronic document creation apparatus 100 based on eye tracking and voice recognition according to an embodiment of the present application (hereinafter, referred to as an 'electronic document creation apparatus 100'), a user 1 Displaying the writing target electronic document 10 to be written, identifying the entry item based on the gaze information of the user 1 on the displayed electronic document 10, and inputting the user 1's voice with respect to the identified entry item The electronic document may be completed in a manner of writing field values corresponding to the entry items based on the .

2 is a schematic configuration diagram of an electronic document creation apparatus based on eye tracking and voice recognition according to an embodiment of the present application.

Referring to FIG. 2 , the electronic document creation apparatus 100 may include a display unit 110 , a gaze analysis unit 120 , and a voice processing unit 130 .

The display unit 110 may display an electronic document including a plurality of entry items. According to one embodiment of the present application, when the creation process of the predetermined electronic document 10 is started, the display unit 110 includes preset writing order information for a plurality of entry items included in the form of the corresponding electronic document 10 . Based on the , it is possible to highlight the entry item corresponding to the first written entry.

According to an embodiment of the present application, the preset writing order information for a plurality of entry items is a predetermined required entry item among a plurality of entry items included in the form of the electronic document 10 displayed by the display unit 110 . It may mean that the writing order is predefined, but is not limited thereto.

3A to 3C illustrate a process of identifying an entry item that the user gazes at based on the user's gaze information with respect to a predetermined electronic document, and entering a field value for the identified entry item based on the user's voice input It is a drawing for

Referring to FIG. 3A , if the form of the electronic document 10 to be written by the user is 'a leave application', the entry item corresponding to the first entry in the preset entry order information for a plurality of entry entries is exemplarily '1' . It may be 'vacation start date and time', which is a sub-item of the 'date and time' label. In this regard, the display unit 110 may display the electronic document corresponding to the 'leave application form', but highlight the entry item for 'vacation start date and time', which is the first written item.

Here, the highlighting of the entry items may be to indicate the entry items that the user stares at or to indicate the order between entry items. Hereinafter, in order to distinguish the area corresponding to a predetermined entry item in order to emphasize the entry item that the user stares at or to indicate the order between the entry items, from the display for indicating the input mode change to be described later, 'first Let's call it 'highlight'. In addition, in contrast to the term of the first highlight display, the display for indicating the input mode change will be referred to as a 'second highlight display'.

The gaze analyzer 120 may acquire the user's gaze information on the displayed electronic document.

Also, the gaze analyzer 120 may identify the entry item that the user gazes at based on the obtained gaze information of the user.

According to an embodiment of the present application, the first highlight display indicating the entry item that the user stares at may be output when the user gazes at an area corresponding to the predetermined entry item for a preset first threshold time or longer. In addition, the first highlight display indicating the order between the entry items may be automatically output for the entry items in the next order when the user completes inputting a field value for a predetermined entry item.

According to one embodiment of the present application, the gaze analysis unit 120 is displayed on the basis of an image including the user's face area photographed by a camera module (not shown) provided in the electronic document creation apparatus 100 , the display unit 110 . In the display area where the electronic document is displayed by the , it may be to identify the position where the user gazes, and to identify the entry item corresponding to the identified position. In other words, the gaze analyzer 120 may include a camera module that captures an image including the user's face region in order to extract the user's gaze information.

As another example, the gaze analyzer 120 may be provided separately from the electronic document creation apparatus 100 to obtain the user's gaze information from an eye tracker device (not shown) worn by the user. For example, the eye tracker device (not shown) may be provided in the form of glasses to be worn by the user, but is not limited thereto.

When the user's gaze information is obtained in connection with the eye tracker device (not shown), the electronic document creation apparatus 100 and the eye tracker device (not shown) may communicate with each other through a network. The network refers to a connection structure in which information exchange is possible between each node, such as terminals and servers. Examples of such a network include a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, and 5G Network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), wifi network, A Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc. are included, but are not limited thereto.

In addition, according to an embodiment of the present application, the gaze analysis unit 120 determines that the user gazes at an area corresponding to a predetermined entry item in the displayed electronic document for a preset threshold time or longer, based on the acquired gaze information of the user. If it is determined, it is possible to switch to an input mode in which a voice input associated with the corresponding entry item can be received.

According to an embodiment of the present disclosure, a preset threshold time for switching to an input mode for a predetermined writing item may be referred to as a second threshold time differentiated from the above-described first threshold time. Here, the second threshold time may be compared with the user's gaze time counted from the point in time when the user gazes at the entry item after the first highlight mark is output for the entry item in order to determine the input mode conversion. In addition, when it is determined by the gaze analysis unit 120 that the user gazes at a predetermined entry item for more than a second threshold time, the display unit 110 displays a second highlight display indicating that the input mode for the entry is started. can be printed out.

In summary, when the user gazes at an entry item for which a predetermined field value has not been input for more than a first threshold time, the entry item is first highlighted, and after the first highlight display for the entry item is output, the user enters the entry When the item is stared at the item for more than the second threshold time, the input mode is switched to the input item, and a second highlight mark may be output. As another example, when the user completes writing a field value for a predetermined entry item, the next entry in the preset writing order is first highlighted, and after the first highlighting of the entry is output, the user When the user gazes at the entry item for more than the second threshold time, it may be switched to an input mode for the entry item and a second highlight mark may be output.

In addition, the electronic document writing apparatus 100 according to an embodiment of the present application provides a sub-order (sub-order) of the non-input mandatory entry item in a state in which a field value for a predetermined required entry item is not input based on preset writing order information. ), when the user stares at the entry for more than the first threshold time, the first highlight for the required entry that the user skips without entering a field value instead of outputting the first highlight for the entry entry to stare at It can operate to display a highlight. In addition, according to the embodiment of the present application, when the first highlighting mark for the skipped mandatory entry item is output, the display unit 110 displays a guide display for allowing the user to recognize that the entry of the field value for the corresponding entry item is omitted. can be printed out.

In addition, according to an embodiment of the present application, in spite of the guidance display output to the user to recognize the omission of the field value for the above-described essential entry item, the user maintains his/her gaze on the entry item corresponding to the lower priority (sub-order). In this case, an input process for the entry item corresponding to the lower priority (lower order) is started, and in this case, the display unit 110 may output a first highlight mark for the entry item corresponding to the lower priority (lower order).

In addition, according to an embodiment of the present application, while an input to the entry item identified based on the user's gaze information (in other words, the entry item switched to the input mode) is being made (during the input mode), the user enters another entry item When the user gazes at for more than the first threshold time, the field value for the entry item that was in the input mode is entered based on the voice input approved by the user until the corresponding point in time (the point in time when it is determined that the other entry item has been gazed at for more than the first threshold time) and a first highlight mark for another entry item that the user stares at for more than a first threshold time may be output, and an input process for another entry item that the user stares at may be started.

In this regard, when the user gazes at an area corresponding to a writing item other than the writing item being written for less than the first threshold time, the electronic document writing apparatus 100 of the present application corresponds to the input mode despite such gaze movement. It is implemented so that the entry item is not changed, so that a short gaze movement of the user that occurs during the input process can be ignored, thereby preventing the input mode of the entry item from being excessively changed according to the gaze movement of the user, and It is possible to create natural electronic documents.

Also, according to an embodiment of the present application, the second threshold time may be set to be greater than the first threshold time. Exemplarily, if the first threshold time is 1 second and the second threshold time is set to 5 seconds, when the user gazes at a predetermined entry item for 1 second or longer, a first highlight is displayed (eg, a shaded indication is generated), and the corresponding entry is displayed. Staring at the item for more than 5 seconds results in a second highlight (eg, the generated shading indicator flashes) and can be switched to input mode. In this regard, the user may visually confirm the second highlight mark and utter a voice input for inputting a field value for the corresponding entry item.

In addition, according to the embodiment of the present application, the first threshold time and/or the second threshold time may be set differently according to each entry item even for the same electronic document. It will be described later in detail with reference to .

In addition, according to an embodiment of the present application, the first highlight mark is to insert a color, shade, pattern, etc. that are distinguished from the rest of the displayed electronic document area in the area corresponding to the predetermined entry item, and the second highlight mark is The color, shade, pattern, etc. inserted in the first highlight display for the corresponding entry item may be displayed so that blinking is repeated.

However, the display types of the first highlight display and the second highlight display are not limited thereto, and the first emphasis is used to induce the user's gaze by reflecting the order between the items to be entered or to make the user recognize the items to be entered by the user. As long as the display and the second highlight mark for recognizing the input item selected for entering a specific field value and converted into an input mode for applying a voice input to the user are recognized separately by the user, the detailed display method is specific not limited in the way

The voice processing unit 130 may receive a user's voice input for the identified entry item. Also, the voice processing unit 130 may write a field value corresponding to the entry item identified based on the received voice input. That is, the voice processing unit 130 may be equipped with a speech-to-text (STT) function to extract a field value from a received voice input and write the extracted field value.

Specifically, according to an exemplary embodiment of the present disclosure, the voice input received from the user includes a portion of an unrefined utterance together with the field value to be actually input with respect to the entry item, or the form of the voice input and the format of the field value required by the entry item. This may be inconsistent. The voice processing unit 130 of the present application may process (normalize) the inputted item by processing (normalizing) the received voice input into a field value refined to conform to the format of the electronic document 10 in consideration of this situation.

Hereinafter, a process in which a voice input received by the voice processing unit 130 is processed (normalized) into a field value actually written in the entry item will be described in detail.

According to an embodiment of the present application, the voice processing unit 130 may determine whether the first segment of the received voice input is a survey and process it. In this case, when the first segment of the received voice input is a survey, the voice processing unit 130 may delete the corresponding segment.

Also, according to an exemplary embodiment of the present disclosure, the speech processing unit 130 may combine segments other than the first segment and perform morphological analysis on the combined segment. Also, the voice processing unit 130 may detect the actual morpheme by searching the morpheme analysis result in the reverse direction.

Also, according to an embodiment of the present disclosure, the voice processing unit 130 may convert the detected real morpheme to conform to a format (part of speech, etc.) required by the corresponding entry item.

For example, the voice processing unit 130 requires that a field value in a noun form be entered in the corresponding entry item, but if the real morpheme detected from the voice input is not in the noun form, it can convert the detected real morpheme into a noun form. can For example, the voice processing unit 130 may convert into the noun form by combining '-ㅁ' or '-sound' after the actual morpheme in the form of a verb (verb, adjective).

As another example, the voice processing unit 130 may convert the detected real morpheme into a date format when it is requested that the field value for the corresponding entry item be input in a date format (eg, year-month-day format). have. Specifically, when the detected real morpheme is 'today', the date of the corresponding day may be converted to conform to the date format and entered.

As another example, if the detected real morpheme is a 'numeric type' and it is requested that a field value for a corresponding entry item also be input in the same numeric type, numeric information itself may be written without additional conversion.

In addition, according to an embodiment of the present application, the format of the corresponding electronic document entry item corresponds to a list-type entry item such as a combo box, a check box, and a radio button (refer to the '2. Category' label in FIG. 3C), but the received If the real morpheme detected from the voice input does not exactly match any one of the preset selection items, the voice processing unit 130 based on the Levenstein edited distance algorithm or the like, the selection item closest to the content included in the detected real morpheme. (Combo box, check box, radio button, etc.) can be selected and assigned as a field value.

Also, the display unit 110 may display a field value processed (normalized) by the voice processing unit 130 and written therein.

In addition, according to an embodiment of the present application, after field values for a predetermined entry item are entered based on a user's voice input through the above-described process, the display unit 110 displays a preset writing order for a plurality of entry items. Based on the information, the entry item corresponding to the next order of the entry entry in which the field value is written may be displayed to be highlighted.

Illustratively, referring to FIG. 3B, '1. When the entry of the field value (refer to FIG. 3b, 'from 09:00 on March 23, 2020') for the entry item corresponding to the sub-item of the date and time label is completed, the display unit 110 may display (first highlighting) to be highlighted (first highlight), which is an entry item corresponding to the next order of 'vacation start date and time', based on preset writing order information.

Also, referring to FIG. 3C , in the case of a list-type entry item in which the user selects a predetermined selection item from a list previously created to include a plurality of selection items such as a combo box, a check box, and a radio button, the display unit 110 ) can be highlighted together with a label indicating a corresponding list-type entry item (refer to FIG. 3C, '2. Classification') and all selection items to allow the user to recognize the types of all selectable selection items.

With respect to such a list-type entry item, the user may select a predetermined selection item by uttering a voice input corresponding to at least one selection item among all displayed selection items.

4 is a diagram for explaining guide information for entry items.

Referring to FIG. 4 , the display unit 110 according to an embodiment of the present application identifies a predetermined writing item within a preset waiting time after it is identified (in other words, after switching to an input mode associated with the predetermined writing item) If a voice input for writing a field value for the entered entry is not received, guide information for the corresponding entry may be displayed.

That is, in the electronic document writing apparatus 100 according to an embodiment of the present application, the user does not utter a voice input for a predetermined entry item even after the switch to the input mode is performed for the predetermined entry item, so that the entry item is pre-recorded. If no input is made for more than the set waiting time, it is determined that the user is having difficulty in inputting (writing) the corresponding entry item, and guide information (refer to FIG. 4 , ' Please provide a brief reason for the vacation.'), including guide information, can be provided to the user.

In this regard, according to an embodiment of the present application, the content of the guide information and the waiting time for displaying the guide information may be determined differently for each format of the electronic document 10 . In addition, the content of the guide information and the waiting time for displaying the guide information may be determined differently for each entry item even for the same electronic document 10 .

Also, according to an embodiment of the present application, the voice processing unit 130 may determine whether the voice input approved by the user is properly input according to the corresponding entry item. In addition, as a result of the determination of the voice processing unit 130, when the user-approved voice input is inappropriate for the entry item (for example, if it includes only content unrelated to the entry item, the entry includes a numerical value that cannot be entered) erroneous input situation), the display unit 110 may display guide information requesting to re-enter the correct voice input for the corresponding entry item.

In addition, according to an exemplary embodiment of the present application with respect to guide information requesting re-ignition in the case of an erroneous input, a predetermined entry item (for convenience of explanation) the voice processing unit 130 is switched to the input mode In the middle of receiving a voice input for 'entry item'), the user gazed at an entry item other than the existing entry item for more than the first threshold time, but at that point (when the user stared at another entry item for more than the first threshold time) If the voice input applied by the user to the existing entry is inappropriate (eg, erroneous input situation, etc.), the input process for the other entry that the user stares at is started. The item movement does not occur, and guide information for requesting re-ignition (re-entry) and erroneous input situation guidance associated with an existing entry item may be displayed. In addition, after the guide information requesting re-ignition is displayed, the input mode for the existing entry item before the gaze movement may be switched again.

According to an embodiment of the present application, the display unit 110 may display (output) guide information in the form of a tooltip, balloon help, etc., but is not limited thereto.

FIG. 5 is a diagram for explaining that a preset threshold time is individually set for each entry item based on the amount of content information associated with the entry item; FIG.

Referring to FIG. 5 , the electronic document creation apparatus 100 according to an embodiment of the present application provides a preset threshold time (first threshold time and / or the second threshold time) can be individually set. Here, the content information associated with the entry item may include information that the user should be familiar with before entering a field value for a predetermined entry item.

Illustratively, referring to FIG. 5 , in the electronic document writing apparatus 100, the user reads the contents of the terms and conditions and field values (' A second threshold time for the corresponding entry item may be set to correspond to the length information of the terms and conditions so that the user can input (spoken) 'agree' or 'disagree'). That is, as the amount of the agreement for obtaining consent increases (the length of the text included in the agreement increases), the second threshold time may be set larger.

For example, the second threshold time is a typical time required to read the corresponding content information based on the amount of content information associated with the corresponding entry item (eg, the average value of the required time collected for a plurality of users). It may be calculated and determined to include a predetermined spare time in the calculated time, but is not limited thereto.

In addition, according to an embodiment of the present application, the electronic document creation apparatus 100 determines whether the user is familiar with the content information of the entry item more accurately, based on the user's gaze information, the region in which the content information is expressed. The degree of movement of the user's gaze in the horizontal direction and the degree of movement in the vertical direction are obtained, and when it is determined that the user has actually read the corresponding content information, switching to the input mode for the corresponding entry item may be performed. For example, the gaze analysis unit 120 displays the content information on the entry item even when it is determined that the user stares at the content information on the entry item for more than the second threshold time based on the acquired gaze information. In the case where it is difficult to see that all content information has been read in light of the degree of movement of the user's gaze in the relevant area compared to the area (area) of the displayed area (gazing at only a specific area that is excessively narrow compared to the total area of the area in which the content information is displayed) In one case, etc.), it is possible to operate so as not to perform the conversion to the input mode for the corresponding entry item.

According to an embodiment of the present application, when the input mode for the entry item for obtaining consent to the terms and conditions is switched, the display unit 110 highlights a check box for selecting consent or non-agreement (eg, blinking, etc.) can be output.

In addition, according to an embodiment of the present application, the voice processing unit 130 utters an utterance including a field name and a field value for a user input item (eg, "{field name} is {field value}", etc.) If the utterance of ) is applied as a voice input, it may operate to write a field value for the entry item corresponding to the field name included in the utterance regardless of the entry item identified (selected) by the user's gaze information. Here, the 'field name' is a representative name representing each entry item, and may be preset for each entry item. For example, 'field name' may be a label name. Also, the voice processing unit 130 may retain synonym information for each of the field names for searching for an utterance part equivalent to a preset field name from the voice input. In this regard, when a utterance part including a field name or a synonym for a field name is detected from the voice input, the voice processing unit 130 may search for a field value in utterances after the detected utterance part.

As an example to help understanding, '1. In the state in which the input mode is switched to the entry item, which is a sub-item of the 'date and time' label, the user speaks a utterance including {field name} and {field value} such as '{vacation class} is {congratulations and congratulations and congratulations}''. If it is applied as input, it is not the entry item in the current input mode, but '2. An exception processing may be performed in which an input to the entry item corresponding to the 'category' label is made.

In addition, according to an embodiment of the present application, when the electronic document creation device 100 receives a user input associated with the creation completion process of the electronic document 10 , the electronic document 10 provides information on required entry items set in advance for the electronic document 10 . It may be determined whether field value writing has been completed. At this time, the display unit 110 of the electronic document writing apparatus 100, when a field value for a predetermined mandatory entry item is omitted, the required input item is displayed as a first highlight to move the gaze to the non-entered required entry item and You can induce input of field values. For example, the electronic document writing apparatus 100 may operate to determine whether writing of all field values for required entry items is completed prior to acquiring the user's signature information.

Also, according to an embodiment of the present application, the voice processing unit 130 may correct (correct) a previously input field value for a predetermined entry item based on a voice input applied by the user in order to correct the field value. . Exemplarily, when the user gazes at an entry item in which a field value has already been inputted based on the gaze information of the user for more than a preset third threshold time, the gaze analysis unit 120 may include information about the entry item (field value). ) to be corrected, output a third highlight for the corresponding entry, and switch to the editing mode, which is a state in which a voice input associated with correction of a field value for the entry can be authorized.

For example, after the voice processing unit 130 is switched to the correction mode for the entry item 'vacation start date and time', when the user applies a voice input for modifying a field value such as 'change it to March 22', the received Based on the voice input, the field value of 'from 09:00 on 2020-03-23' may be corrected (corrected) to 'from 09:00 on March 22, 2020'.

According to an exemplary embodiment of the present disclosure, the third threshold time may be determined to be the same as the second threshold time for switching the input mode (including those that can be viewed as substantially equivalent). As another example, the third threshold time associated with the correction of the field value may be determined to be larger than the second threshold time so that the correction of the field value can be performed more carefully.

Also, according to an exemplary embodiment of the present disclosure, display methods of colors, shades, patterns, etc. for the second highlight display corresponding to the input mode and the third highlight display corresponding to the correction mode may be different, but the present disclosure is limited thereto. it is not

In addition, according to an embodiment of the present application, the voice processing unit 130 selects {field name} and {field value} to be corrected, such as 'Change the vacation start date and vacation end date to March 22,' when the voice input associated with the correction is In the case of all-inclusive form, regardless of the user's gaze information, the input item corresponding to {field name} may be operated to be replaced with {field value} to be corrected.

In addition, according to an embodiment of the present application, the display unit 110 displays that the field value is corrected when the previously input field value is corrected based on the voice input applied in the correction mode, and the user confirms the field value. guide information for receiving again may be displayed. Illustratively, the guide information for receiving the confirmation input includes a button for confirming and applying correction of a field value (eg, 'confirm', 'yes', 'YES', etc.) and a button for canceling correction of a field value (eg, 'cancel', 'no', 'NO', etc.) may be displayed in the form of a pop-up window.

Also, according to an embodiment of the present application, the confirmation input related to the guide information displayed in connection with the correction of the above-described field value may be applied in the form of a voice input. In other words, by way of example, a voice input corresponding to each of the buttons displayed in the guide information for receiving the confirmation input (eg, utterances of confirmation, yes, YES, no, NO, cancel, etc.) When the user approves, the correction (correction) to the previously input field value may be definitively applied or the correction (correction) may not be reflected.

In addition, the storage unit (not shown) of the electronic document writing apparatus 100 completes the entry of field values for all entry items included in the electronic document or the field values for some required entry items according to the format of the electronic document. When the writing is completed and the creation of the electronic document is finished, the completed electronic document may be stored.

According to an embodiment of the present application, when the electronic document is stored by the storage unit (not shown), the display unit 110 may display a summary screen and/or a detailed screen of the completed electronic document.

In addition, the signature obtaining unit (not shown) of the electronic document creation apparatus 100 may receive a handwritten signature input from the user with respect to the completed electronic document. In this regard, the storage unit (not shown) of the electronic document creating apparatus 100 may store the completed electronic document including handwritten signature information.

According to an embodiment of the present disclosure, the handwritten signature information may include a detection result including at least one of handwriting information or pressure information of a user detected in a handwritten signature input process and the shape of the handwritten signature itself.

In addition, according to an embodiment of the present application, the signature obtaining unit (not shown) is stored in the handwritten signature written before the creation of the electronic document for the corresponding user (or the user's account), the stored handwritten signature exists. You can call up your signature and write it in the signature field of the e-document. In addition, according to an embodiment of the present application, if the user's written handwritten signature does not exist, the display unit 110 displays a signature input window requesting the user to write a new handwritten signature, and the signature obtaining unit ( (not shown) may acquire handwritten signature information based on a user input (eg, a touch input through a finger, a dedicated input pen, etc.) applied to the displayed signature input window.

Hereinafter, an operation flow of the present application will be briefly reviewed based on the details described above.

6 is an operation flowchart for a method of creating an electronic document based on eye tracking and voice recognition according to an embodiment of the present application.

The method for creating an electronic document based on eye tracking and voice recognition illustrated in FIG. 6 may be performed by the electronic document writing apparatus 100 described above. Accordingly, even if omitted below, the description of the electronic document creation apparatus 100 may be equally applied to the description of the gaze tracking and voice recognition-based electronic document creation method.

Referring to FIG. 6 , in step S11 , the display unit 110 may display an electronic document including a plurality of entry items.

Also, in step S11 , the display unit 110 may display the entry item corresponding to the first entry item to be highlighted based on preset writing order information for the plurality of entry items. Here, the writing order information may mean that a writing order for a predetermined required writing item among a plurality of writing items included in the displayed electronic document is defined in advance.

Next, in step S12 , the gaze analyzer 120 may acquire the gaze information of the user on the displayed electronic document.

Next, in step S13 , the gaze analyzer 120 may identify the entry item that the user gazes at based on the obtained gaze information of the user.

Specifically, in step S13, when it is determined that the user gazes at an area corresponding to a predetermined entry item in the displayed electronic document for more than a preset threshold time, the gaze analysis unit 120 receives a voice input associated with the entry item You can switch to the input mode that is possible.

Next, in step S14 , the voice processing unit 130 may determine whether the user's voice input for the identified entry item has been received within a preset waiting time.

If the voice input is not received within a preset waiting time after switching to the input mode in step S14, the display unit 110 may display guide information for the corresponding entry item in step S141.

Next, in step S15 , the voice processing unit 130 may receive a user's voice input for the identified entry item.

Next, in step S16 , the voice processing unit 130 may write a field value corresponding to the entry item identified based on the received voice input.

Next, in step S17 , the display unit 110 displays the entry items corresponding to the next order of the entry items in which field values are written in step S16 to be highlighted based on the preset writing order information for the plurality of entry items. can

In the above description, steps S11 to S17 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted if necessary, and the order between the steps may be changed.

The method for creating an electronic document based on eye tracking and voice recognition according to an embodiment of the present application may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the above-described method for creating an electronic document based on eye tracking and voice recognition may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The above description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

100: Eye tracking and voice recognition-based electronic document creation device
110: display unit
120: gaze analysis unit
130: voice processing unit
10: Electronic document

Claims (11)

As an electronic document creation method based on eye tracking and voice recognition,
displaying an electronic document including a plurality of entry items;
obtaining information about a user's gaze on the displayed electronic document;
identifying a writing item that the user gazes at based on the gaze information;
receiving the user's voice input for the identified entry item; and
filling in a field value corresponding to the identified entry item based on the received voice input;
including,
The step of identifying the entry is:
when it is determined that the user gazes at a region corresponding to a predetermined entry item in the displayed electronic document for a preset first threshold time or longer, outputting a first highlight mark for the region; and
If it is determined that the user continues to stare at the area for more than a preset second threshold time after the first highlight display is output, the input mode is switched to an input mode in which the voice input associated with the entry item can be received, outputting a second highlight for the corresponding area;
including,
The second threshold time is set to be larger than the first threshold time, and the first and second highlight marks are output in different ways so that the user can recognize them separately. delete The method of claim 1,
After entering the field value,
displaying an entry item corresponding to an order next to the entry item in which the field value is written to be highlighted based on preset writing order information for the plurality of entry items;
Which will further include, the electronic document creation method. 4. The method of claim 3,
The writing order information is
The method of creating an electronic document in which a writing order for a predetermined mandatory entry item among a plurality of entry items included in the displayed electronic document is predefined. 4. The method of claim 3,
The step of displaying the electronic document comprises:
displaying an entry item corresponding to the first written item to be highlighted based on the writing order information;
That comprising a, electronic document creation method. According to claim 1,
After identifying the entry,
If the voice input is not received within a preset waiting time after switching to the input mode, displaying guide information for a corresponding entry item;
Which will further include, the electronic document creation method. As an electronic document creation device based on eye tracking and voice recognition,
a display unit for displaying an electronic document including a plurality of entry items;
a gaze analyzer configured to obtain gaze information of a user on the displayed electronic document, and identify an entry item that the user gazes at based on the gaze information; and
a voice processing unit that receives the user's voice input for the identified entry item, and writes a field value corresponding to the identified entry entry based on the received voice input;
including,
The gaze analysis unit,
When it is determined that the user gazes at the area corresponding to the predetermined entry item in the displayed electronic document for a preset first threshold time or longer, a first highlight mark is output for the area, and the first highlight mark is outputted After that, if it is determined that the user continues to stare at the corresponding area for more than a preset second threshold time, the input mode is switched to an input mode in which the voice input associated with the corresponding entry item can be received, and a second emphasis is displayed on the corresponding area output, but
The second threshold time is set to be larger than the first threshold time, and the first and second highlight marks are output in different ways so that the user can distinguish and recognize them. delete 8. The method of claim 7,
The display unit,
and displaying an entry item corresponding to an order next to the entry item in which the field value is written to be highlighted based on preset entry order information for the plurality of entry items. 10. The method of claim 9,
The display unit,
and displaying an entry item corresponding to an initially created item to be emphasized based on the writing order information. 8. The method of claim 7,
The display unit,
If the voice input is not received within a preset waiting time after switching to the input mode, guide information for the corresponding entry item is displayed.

Images