JP2015212981A - Electronic device, electronic document control program and electronic document control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a user to go on reading an electronic document without performing complicated operations just by browsing the electronic document.SOLUTION: An electronic document terminal 10 includes a display 26 for displaying the electronic document and a camera module for detecting a line-of-sight position (EP) of a user. For instance, in the displayed electronic document, a detection area (DA) where sentences and images are displayed is included. Also, on the basis of the detected line-of-sight position (EP), a line-of-sight speed (ES) and staying time (ET) during which the line-of-sight position (EP) stays in the detection area (DA) or the like are obtained. Further, when the line-of-sight speed (ES) is equal to or lower than a first threshold and the staying time (ET) is equal to or longer than a second threshold, it is determined that the detection area in the electronic document is read. For instance, when it is determined that the electronic document of a cartoon is read, a dialogue is displayed in a balloon (F) or the next page is displayed.

Description

  The present invention relates to an electronic device, an electronic document control program, and an electronic document control method, and more particularly to an electronic device, an electronic document control program, and an electronic document control method that display an electronic document.

An example of an electronic device that displays an electronic document is disclosed in Patent Document 1. In the electronic book device of Patent Document 1, it is possible to detect the user's line of sight and determine the position of interest on the display unit. When an image based on the book data is displayed on the display unit, the attention part in the book data displayed at the attention position is specified, and the content represented by the attention part is highlighted.
JP 2007-102360 A [G06F 3/048, G06F 17/30]

  However, in the electronic book device of Patent Document 1, it cannot be determined whether the user understands the contents of the book data.

  Therefore, a main object of the present invention is to provide a novel electronic device, electronic document control program, and electronic document control method.

  Another object of the present invention is to provide an electronic device, an electronic document control program, and an electronic document control method that allow a user to grasp the contents of the electronic document.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate the corresponding relationship with the embodiments described in order to help understanding of the present invention, and do not limit the present invention.

  1st invention is the electronic device which has a display part which displays an electronic document, Comprising: The imaging | photography part which image | photographs an image, The gaze position which detects the gaze position with respect to a display part based on the image image | photographed by the imaging | photography part When the determination unit determines whether the electronic document displayed on the display unit is read based on the line-of-sight position detected by the detection unit, the line-of-sight position detection unit, and when the electronic document is read by the determination unit, An electronic apparatus includes an execution unit that executes predetermined processing related to an electronic document.

  In the first invention, the display unit (26) of the electronic device (10: reference numerals exemplifying corresponding parts in the embodiment, hereinafter the same) is, for example, an LCD monitor, and is a contract, novel, comic book and picture book. Display an electronic document. The imaging unit (36-40) captures an image of the user, for example, and the line-of-sight position detection unit (20, S99) detects the user's line-of-sight position (EP) based on the image. The determination unit (20, S49-S53) determines whether the displayed electronic document has been read based on the detected movement of the line-of-sight position. When it is determined that the electronic document has been read, the execution unit (20, S3, S55, S63, S71, S81) executes a predetermined process of displaying the next page or displaying additional information.

  According to the first invention, the user can read the electronic document without performing complicated operations only by browsing the electronic document.

  A second invention is according to the first invention, further comprising a gaze speed calculation unit that calculates a gaze movement speed based on the gaze position detected by the gaze position detection unit, and the determination unit calculates the gaze speed When the line-of-sight speed calculated by the unit is equal to or less than the first threshold, it is determined that the electronic document displayed on the display unit has been read.

  In the second invention, the line-of-sight speed calculation unit (20, S101) calculates the line-of-sight speed (ES) every predetermined time based on the change in the line-of-sight position for each frame time. If the line-of-sight speed at the line-of-sight position moves below the first threshold (THa), it is determined that the displayed electronic document has been read.

  According to the second invention, it is possible to determine whether or not the content of the electronic document has been read by using the gaze speed based on the gaze position.

  A third invention is according to the first invention, wherein the electronic document includes a detection area for displaying a sentence or an image, and the time when the line-of-sight position detected by the line-of-sight position detection unit stays in the detection area is determined. A stay time measuring unit for measuring is further provided, and the determination unit determines that the electronic document displayed on the display unit has been read when the stay time measured by the stay time measurement unit is equal to or greater than the second threshold.

  In the third invention, the electronic document includes a detection area (DA) in which a sentence or an image is displayed. The stay time measuring unit (20, S105) measures the stay time (ET) in which the detected line-of-sight position stays in the detection area. If the stay time of the line-of-sight position in the detection area is equal to or greater than the second threshold (THb), it is determined that the displayed electronic document has been read.

  According to the third invention, it is possible to determine whether the content of the electronic document has been read using the staying time based on the line-of-sight position.

  A fourth invention is according to the first invention, and the electronic document includes a detection area for displaying a sentence or an image, and calculates a movement speed of the line of sight based on the line-of-sight position detected by the line-of-sight position detection unit. The line-of-sight calculation unit and the line-of-sight position detected by the line-of-sight position detection unit further includes a stay time measurement unit that measures the time during which the line-of-sight position stayed in the detection region, When the speed is equal to or lower than the first threshold and the stay time measured by the stay time measuring unit is equal to or greater than the second threshold, it is determined that the electronic document displayed on the display unit has been read.

  In the fourth invention, the electronic document includes a detection area (DA) in which a sentence or an image is displayed. The line-of-sight speed calculation unit (20, S101) calculates the line-of-sight speed (ES) every predetermined time based on the change in the line-of-sight position for each frame time. The stay time measuring unit (20, S105) measures the stay time (ET) in which the detected line-of-sight position stays in the detection area. Then, if the line-of-sight speed of the line-of-sight position moves below the first threshold (THa) and the staying time of the line-of-sight position in the detection area is above the second threshold (THb), it is determined that the displayed electronic document has been read. The

  According to the fourth invention, it is possible to determine whether or not the content of the electronic document has been read using the line-of-sight speed and the staying time based on the line-of-sight position.

  The fifth invention is dependent on any one of the first to fourth inventions, and when it is determined that the end operation has been performed by the end operation determining unit for determining whether the end operation has been performed, and the end operation determining unit. And a storage unit that stores interruption data based on the line-of-sight position detected by the line-of-sight position detection unit.

  In the fifth invention, the end operation determination unit (20, S59) determines whether or not an end operation for finishing the display of the electronic document has been performed. When the display of the electronic document is finished, the storage unit (20, S61) stores the detection area including the line-of-sight position, the page number to be displayed, and the like as interruption data.

  According to the fifth aspect, even if the user interrupts the browsing of the electronic document, the browsing of the electronic document can be easily resumed by reading the interruption data.

  A sixth invention is dependent on the first to fifth inventions, wherein the electronic document includes a plurality of pages, and when the execution unit determines that one page of the electronic document has been read, the next page Is further included.

  In the sixth invention, the electronic document includes a plurality of pages. For example, when it is determined that a certain page has been read, the next page display unit (20, S3) displays the next page.

  According to the sixth invention, the user can display the next page only by reading the sentence.

  The seventh invention is dependent on any one of the first to sixth inventions, and the execution unit agrees or disagrees with the contents of the electronic document when the determination unit determines that the electronic document has been read. A key display unit that displays keys for operation.

  In the seventh invention, the electronic document includes, for example, a contract. If it is determined that the contract has been read, the key display unit (20, S63) displays a key for accepting or disagreeing with the contents of the contract in an operable manner. The user uses these keys to agree or disagree with the contents of the contract.

  According to the seventh aspect of the invention, it is possible to prevent the user from consenting without grasping the contents of the electronic document or consenting from an erroneous operation.

  The eighth invention is dependent on any of the third to seventh inventions, and when the execution unit determines that the detection region of the electronic document has been read by the determination unit, the execution unit adds additional information to the detection region. A first additional information display unit for displaying is further included.

  In the eighth invention, for example, the electronic document includes a comic, and the detection area includes a comic frame. Moreover, a 1st additional information display part (20, S71) displays a dialog as additional information in the balloon (F) contained in a comic frame, when the detection area | region containing a comic frame is read.

  According to the eighth aspect, it is possible to assist reading of the electronic document by displaying additional information after it is determined that a part of the electronic document has been read.

  A ninth invention is dependent on any one of the third to eighth inventions, and when the execution unit determines that the detection region in which the text included in the electronic document is displayed is read by the determination unit, It further includes a second additional information display unit that displays additional information in a detection area different from the detection area where the text is displayed.

  In the ninth invention, the electronic document includes, for example, a picture book. For example, when the second additional information display unit (20, S81) determines that the detection area in which the dialogue is written has been read, an image (picture) corresponding to the dialogue is displayed in the other detection areas as additional information. The

  According to the ninth aspect of the invention, a user who reads an electronic document can be surprised by a visual change.

  The tenth aspect of the invention is an image obtained by photographing a processor (20) of an electronic device (10) having a display unit (26) for displaying an electronic document and a photographing unit (36-40) for photographing an image. The line-of-sight position detection unit (S99) for detecting the line-of-sight position relative to the display unit, and the determination for determining whether the electronic document displayed on the display unit has been read based on the line-of-sight position detected by the line-of-sight position detection unit Electronic document control that functions as an execution unit (S3, S55, S63, S71, S81) that executes predetermined processing related to the electronic document when the electronic document is determined to be read by the units (S49-S53) and the determination unit It is a program.

  In the tenth invention as well, as in the first invention, the user can read the electronic document without complicated operations just by browsing the electronic document.

  An eleventh aspect of the invention is an electronic document control method for an electronic device (10) having a display unit (26) for displaying an electronic document and a photographing unit (36-40) for photographing an image, and is photographed by the photographing unit. The line-of-sight position with respect to the display unit is detected based on the image (S99), and it is determined whether the electronic document displayed on the display unit has been read based on the detected line-of-sight position (S49-S53). When it is determined that the document has been read, a predetermined process related to the electronic document is executed (S3, S55, S63, S71, S81).

  In the eleventh aspect of the invention, the user can read the electronic document without performing complicated operations only by browsing the electronic document.

  According to the present invention, the user can read the electronic document without complicated operations only by browsing the electronic document.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

FIG. 1 is an illustrative view showing an electrical configuration of an electronic document terminal according to an embodiment of the present invention. FIG. 2 is an illustrative view showing one example of an appearance of the electronic document terminal shown in FIG. FIG. 3 is an illustrative view showing one example of a line-of-sight position detected by the electronic document terminal shown in FIG. FIG. 4 is an illustrative view showing one example of a change in line-of-sight position detected by the electronic document terminal shown in FIG. FIG. 5 is an illustrative view showing another example of a change in the line-of-sight position detected by the electronic document terminal shown in FIG. FIG. 6 is an illustrative view showing one example of an electronic document displayed on the display shown in FIG. FIG. 7 is an illustrative view showing one example of a configuration of the electronic document shown in FIG. FIG. 8 is an illustrative view showing one example of a change in the line-of-sight position detected in the electronic document shown in FIG. FIG. 9 is an illustrative view showing another example of the electronic document shown in FIG. FIG. 10 is an illustrative view showing one example of a state table stored in the RAM shown in FIG. FIG. 11 is an illustrative view showing another example of an electronic document displayed on the display shown in FIG. FIG. 12 is an illustrative view showing another example of the electronic document displayed on the display shown in FIG. FIG. 13 is an illustrative view showing still another example of the electronic document displayed on the display shown in FIG. FIG. 14 is an illustrative view showing one example of a memory map of the RAM shown in FIG. FIG. 15 is a flowchart showing an example of the electronic document control process of the processor shown in FIG. FIG. 16 is a flowchart showing an example of text display processing of the processor shown in FIG. FIG. 17 is a flowchart showing an example of the text / image display processing of the processor shown in FIG. FIG. 18 is a flowchart showing an example of image display processing of the processor shown in FIG. FIG. 19 is a flowchart showing an example of the line-of-sight position detection processing of the processor shown in FIG.

  Referring to FIG. 1, an electronic document terminal (also referred to as an electronic book terminal or an electronic book reader) 10 of this embodiment is a kind of electronic device or portable device, and “electronic documents” such as novels, comics, picture books, and the like. Display for browsing. Further, the electronic document terminal 10 performs a process of turning a page of the electronic document being displayed in accordance with a user operation. The “electronic document” in this embodiment includes magazines, newspapers, contracts, and the like in addition to books such as novels, comics, picture books, photo books, and instruction manuals.

  The electronic document terminal 10 includes a processor 20 called a computer or CPU. A key input device 22, a display driver 24, a flash memory 28, a RAM 30, a touch panel control circuit 32 and a camera control circuit 36 are connected to the processor 20. The display driver 24 is connected to a display 26 that functions as a display unit. Further, a touch panel 34 is connected to the touch panel control circuit 32. A control motor (not shown) for controlling the lens positions of the image sensor 38 and the focus lens 40 is connected to the camera control circuit 36.

  The processor 20 manages the overall control of the electronic document terminal 10. The RAM 30 is used as a work area (including a drawing area) or a buffer area of the processor 20. Content data such as characters, images, and electronic documents of the electronic document terminal 10 are recorded in the flash memory 28.

  The key input device 22 includes operation keys and function keys. Then, information (key data) on keys operated by the user is input to the processor 20. Further, when each key included in the key input device 22 is operated, a click sound is generated. Therefore, the user can obtain an operational feeling for the key operation by listening to the click sound.

  The display driver 24 controls display on the display 26 connected to the display driver 24 under the instruction of the processor 20. The display driver 24 includes a video memory (not shown) that temporarily stores image data to be displayed. Although not shown, the display 26 is illuminated by a backlight.

The touch panel 34 is a capacitance type that detects a change in capacitance between electrodes caused by an object such as a finger approaching the surface. For example, it detects that one or more fingers touch the touch panel 34. To do. The touch panel 34 is a pointing device that is provided on the display 26 and indicates an arbitrary position within the screen. The touch panel control circuit 32 detects a touch operation that is pressed, stroked, or touched within the effective touch range of the touch panel 34, and outputs coordinate data indicating the position of the touch operation to the processor 20. That is, the user can input an operation direction, a figure, or the like to the electronic document terminal 10 by pressing, stroking, or touching the surface of the touch panel 34 with a finger.

  Here, an operation in which the user touches the upper surface of the touch panel 34 with a finger is referred to as “touch”. On the other hand, an operation of releasing a finger from the touch panel 34 is referred to as “release”. Then, an operation in which the user touches the upper surface of the touch panel 34 and continuously releases is referred to as “touch and release”.

  Further, the operation of stroking the surface of the touch panel 34 is referred to as “slide”, and the operation of sliding after touching and releasing is referred to as “touch slide”. Furthermore, the operation of performing touch and release twice in succession is referred to as “double tap”, and the operation of touching two locations substantially simultaneously is referred to as “multi-touch”. The “touch operation” includes the above touch, release, touch and release, slide, touch slide, double tap, multi-touch, and the like.

  The coordinates indicated by the touch are referred to as “touch points” (touch start positions), and the coordinates indicated by the releases are referred to as “release points” (touch end positions).

  Note that the touch operation is not limited to a finger, and may be performed with a touch pen with a conductor attached to the tip. Further, as a detection method of the touch panel 34, a surface-type capacitance method may be employed, or a resistance film method, an ultrasonic method, an infrared method, an electromagnetic induction method, or the like may be used.

  The camera control circuit 36, the image sensor 38, the focus lens 40, and the like may be collectively referred to as a camera module or a photographing unit. The camera control circuit 36 is a circuit for outputting a still image or a moving image with the mobile phone 10. In the imaging area of the image sensor 38, a light receiving element corresponding to SXGA (1280 × 1024 pixels) is displayed. For this reason, when the image sensor 38 is irradiated with the optical image of the subject, a charge corresponding to the optical image of the subject, that is, a SXGA raw image signal is generated by photoelectric conversion in the imaging area. The user can change the image size (number of pixels) of the image data to XGA (1024 × 768 pixels), VGA (640 × 480 pixels), etc. in addition to SXGA.

  For example, when the camera function is executed, the processor 20 activates an image sensor driver built in the camera control circuit 36 and instructs the image sensor driver to perform an exposure operation and a charge readout operation corresponding to a designated readout region. To do.

  The image sensor driver performs exposure of the imaging surface of the image sensor 38 and reading of electric charges generated by the exposure. As a result, a raw image signal is output from the image sensor 38. The output raw image signal is input to the camera control circuit 36. The camera control circuit 36 performs processing such as color separation, white balance adjustment, and YUV conversion on the input raw image signal, and the YUV format. Generate image data. Then, YUV format image data is input to the processor 20.

The YUV format image data input to the processor 20 is received by the processor 20 as RA.
Stored (temporarily stored) in M30. Here, when the camera function is executed to detect the user's line-of-sight position, the line-of-sight position is detected based on the YUV format image data stored in the RAM 30. Since the detection of the line-of-sight position will be described later, a detailed description thereof is omitted here. The line-of-sight position referred to here means a position that the user is paying attention to based on the display range of the display 26.

The camera control circuit 36 calculates a focus evaluation value from the raw image signal and outputs the focus evaluation value to the processor 20. The processor 20 executes an AF (Autofocus) process based on the focus evaluation value output from the camera control circuit 36. When the AF process is executed, the camera control circuit 36 adjusts the lens position of the focus lens 40 by controlling the lens motor under the instruction of the processor 20. As a result, an image in which the subject is in focus is captured.

  If the setting is made to display the photographed image on the display 26, the stored YUV format image data is converted into RGB format image data by the processor 20 and then given from the RAM 30 to the display driver 28. It is done. Then, RGB format image data is output to the display 26. As a result, a through image of low pixels (for example, 320 × 240 pixels) representing the subject is displayed on the display 26.

  FIG. 2 is an external view showing the external appearance of the electronic document terminal 10. Referring to FIG. 2, the electronic document terminal 10 has a planar rectangular casing C. The key input device 22 includes a first operation key 22a, a second operation key 22b, and a function key 22c. These keys are provided on the surface of the casing C. The display 26 is attached so that the display surface (monitor screen) can be seen from the surface of the housing C. A touch panel 34 is provided on the display surface of the display 26.

  For example, the user operates the function key 22c to display a list of electronic document files on the display 26, and selects an arbitrary electronic document file by a touch operation. When the electronic document is displayed, the pages of the electronic document can be turned (changed) using the first operation key 22a and the second operation key 22b. For example, when the first operation key 22a is operated, the previous page is displayed, and when the second operation key 22b is operated, the next page is displayed.

  Further, the opening OP1 communicates with the focus lens 40 and the image sensor 38. That is, a camera module (not shown) is built in the housing C, and images the object scene on the surface side of the housing C through the opening OP1. The opening OP1 is provided on one surface in the longitudinal direction of the housing C so that the user's face can be photographed.

  Here, when detecting a user's gaze position, a gaze position is detected based on the user's face image image | photographed as FIG. 3 (A). More specifically, the processor 20 determines whether a face area is included in an image (captured image) output from the camera module. Further, if the photographed image includes a face area, an eye area is extracted. Further, when the eye area is extracted, predetermined image processing is applied, and the center of the black eye area is calculated to estimate the user's line-of-sight direction. Then, as shown in FIG. 3B, the line-of-sight position EP is detected based on the line-of-sight direction and the line-of-sight distance between the user's face and the display 26.

  Note that the line-of-sight distance may be set in advance as a value obtained from an experiment or the like, or may be estimated from the size of the face area of the user in the captured image. Furthermore, when a distance measurement sensor using infrared rays or the like is provided, the line-of-sight distance may be obtained using the measurement sensor.

  In addition, since the processor 20, the display driver 24, the flash memory 28, the RAM 30, the touch panel control circuit 34, the camera control circuit 36, the image sensor 38, and the focus lens 40 are built in the housing C, FIG. 2 and FIG. ), (B), not shown.

  Next, in this embodiment, the line-of-sight movement speed (line-of-sight speed) ES based on the detected line-of-sight position, and the time (stay time) ET that the line-of-sight position stays in the detection area DA such as a picture or a sentence in the electronic document And calculate. Hereinafter, the line-of-sight speed ES and the staying time ET will be described, where T is a certain time, and Δt [ms] is a time (frame time) when the display 26 is updated.

First, the line-of-sight speed ES will be described. Figure 4 (A) - (D) is a display area of the display 26 shows the change of the line-of-sight position EP of each Delta] t [ms] from a certain time T _1. For example, from time T ₁ when Delta] t [ms] has elapsed, the line-of-sight position EPa has moved to the line-of-sight position EPb, the its distance two points (variation) is d1 [mm]. Further, when the time T ₁ 2Δt [ms] has elapsed, the line-of-sight position EPb has moved to the line-of-sight position EPc, change amount is d2 [mm]. Further, when the time T ₁ 3Δt [ms] has elapsed, the line-of-sight position EPc has moved to the line-of-sight position EPd, variation becomes d3 [mm].

In this embodiment, the instantaneous speed s _m [mm / ms] for each frame is calculated based on the frame time Δt [ms] and each change amount d [mm], and further, for a predetermined time (for example, 3 Δt [ms]). The average of the instantaneous velocities s _m [mm / ms] is calculated as the visual line speed ES. Therefore, line-of-sight speed ES
Is calculated (updated) every predetermined time.

Next, the stay time T will be described. Figure 5 (A) - (D) is in the display area of the display 26, and from a certain time T _2, shows a change of the line-of-sight position EP of each Delta] t [ms]. In addition, although a sentence and an image are omitted, a detection area DA is provided in the display area. For example, the line-of-sight position EPe user is not included in the time T _2, the detection region DA, contains viewpoint position EPf at a later time Delta] t seconds (T 2 ₊ Δt) in the detection area DA. Although the time _(T 2 + _2Δt) any line-of-sight position EPg is included in the detection area DA, a time _(T 2 + _3Δt) in line-of-sight position EPh is out of the detection area DA. In other words, in between time _{T 2,} the time of _(T 2 + _3Δt), from the time _{(T 2} + Δt) to the time _(T 2 + _3Δt), because it contains viewpoint position EPg the detection area DA, gaze in the detection area DA The stay time ET at the position EP is “2Δt [ms]”. Further, as a specific measurement method, it is determined whether the line-of-sight position EP is included in the detection area DA every time the frame is updated, and the number of times that it is continuously determined that it is included is counted. The product of the counted number and the frame time Δt is the stay time ET.

  Note that the calculation method and measurement method of the line-of-sight speed ES and the stay time ET are not limited to the above-described methods. In other embodiments, the line-of-sight speed ES and the stay time ET may be obtained using other methods. Good.

  FIG. 6 is an illustrative view showing an example of a display 26 on which an electronic document is displayed. The display 26 includes a status display area 60 for displaying icons representing the date and time, the remaining battery level, and the like, and a function display area 62 for displaying an electronic document. Here, since an operation of browsing the electronic document file of the contract is performed by the user, the contract is displayed in the function display area 62 after the metadata described later is read.

Referring to FIG. 7A, the electronic document (contract) includes a detection area DAa including a title sentence, a detection area DAb-DAd including a sentence indicating contents, and the like.
A consent key 70 for agreeing to the contents of the contract and a disagreement key 72 for disagreeing are also included. Further, in the initial state, the operations on the consent key 70 and the disagreement key 72 are invalidated. In order to represent such a state, the two keys are surrounded by a dotted frame and the font of the character is italicized. Has been.

  Further, the metadata of the electronic document file includes the file name, type, number of pages, and coordinate range of the detection area DA of the electronic document, and a first threshold value for determining whether to execute a predetermined process, which will be described later, The second threshold, order, start area, etc. are also included.

  For example, referring to FIG. 7B, in the metadata of the contract (electronic document) file, “Terms of Use” is recorded as the file name, and the electronic document is mainly composed of text as the type. Is recorded, and a coordinate range such as detection areas DAa-DAd is recorded in the detection area. In the number of pages, “1 page” is recorded as the total number of pages of the contract.

Further, in the metadata, a threshold value THa [mm / ms] is recorded as the first threshold value of the line-of-sight velocity, and a threshold value THb [ms] is recorded as the second threshold value of the staying time. Furthermore, “DAa, DAb, DAc, DAd,...” Is recorded as the order of browsing the detection area DA, and “DAa” is recorded as the position for starting control based on the line-of-sight position EP in the start area. The

  Here, in this embodiment, it is determined whether the line-of-sight speed ES and the stay time ET based on the user's line-of-sight position EP satisfy the conditions based on the first threshold value and the second threshold value in all the detection areas DA. When this condition is satisfied, it is determined that the content of the detection area DA has been read by the user, and predetermined processing is executed according to the type of electronic document.

  More specifically, first, when the metadata of the electronic document is read by the processor 20 and the first threshold value and the second threshold value are recorded, it is determined that it is necessary to execute a predetermined process based on the line of sight.

  With reference to FIG. 8A, when the user turns his / her line of sight to the detection area DAa and the line-of-sight position EPj is detected in the detection area DAa in the state determined as described above, the start area in this electronic document is displayed. Is determined. Here, since the detection area DAa including the line-of-sight position EPj is the start area, the type of “sentence” is read from the metadata, and the document display process is executed corresponding to “sentence”.

  When the document display process is executed, it is determined whether or not the line-of-sight speed ES is equal to or less than the first threshold value THa, and the stay time ET of the line-of-sight position EP in the detection area DAa is equal to or greater than the second threshold value THb. In this state, when the user moves from the line-of-sight position EPj to the line-of-sight position EPk and the above condition is satisfied, it is determined that the contents of the detection area DA have been read. Then, as shown in FIG. 8B, the already-read process is executed in which the typeface of the character included in the detection area DAa is changed to italic and the character size is reduced.

  As described above, in this embodiment, it is possible to determine whether the content of the electronic document has been read using the line-of-sight speed ES based on the line-of-sight position EP and the stay time ET in the detection area DA that displays characters.

Furthermore, the processor 20 determines whether or not the line-of-sight speed ES and the staying time ET satisfy the conditions based on the first threshold value THa and the second threshold value THb for the other detection areas DA. Then, referring to FIG. 9, when it is determined that the contents of all detection areas DA have been read by the user, a read-out process is executed as a predetermined process so that consent key 70 and disagreement key 72 can be selected. . That is, since it can be determined that the contents of the contract displayed as an electronic document have been completely read by the user, the consent key 70 and the disagreement key 72 are selected so that the user can agree or disagree with the contract contents. It is made into a state. Note that these keys may be operated by a touch operation on the touch panel 34, or may be operated by the first function key 22a and the second function key 22b.

  As described above, in this embodiment, when the contract is displayed as an electronic document so that it can be browsed, the user can read the content and then agree or disagree with the content. Thereby, it can prevent that a user agrees without grasping the contents of a contract, or agrees by mistaken operation.

When the line-of-sight position EP is detected, the detection result is recorded in the state table shown in FIG. Referring to FIG. 10, the state table includes the detected current gaze position EP coordinates, the calculated current gaze speed, the calculated current stay time, and the current gaze position EP. The area, the history of the detection area DA determined to have been read, and the currently displayed page number are recorded. For example, as shown in FIG. 8B, when the user's line-of-sight position EP moves and the line-of-sight position EPm is included in the detection area DAb, “Ex, Ey” is recorded as the line-of-sight position EP in the state table. “ES _n [mm / ms]” is recorded as the line-of-sight velocity ES, “ET _n [ms]” is recorded as the stay time, “DAb” is recorded as the current area, and “DAa” is recorded as the area history. “1/1” indicating that the first page among all the pages is being recorded is recorded.

  Further, when an end operation is performed in which the function key 22c is pressed and held while browsing the electronic document, the current area, area history, page, and the like recorded in the state table are stored as interruption data. That is, even if the user interrupts browsing of the electronic document, browsing of the electronic document can be easily resumed by using the interruption data as a “bookmark”.

  Here, in addition to the contract, a “novel” mainly composed of text as in the contract, a “cartoon” in which text and images are included in one detection area DA, and mainly composed of images Even when an electronic document such as “picture book” is displayed, the predetermined processing is executed based on the line-of-sight position EP.

  Referring to FIGS. 11A and 11B, when a novel whose type of electronic document is “sentence” is displayed, for example, a detection area DA is set for each sentence. For example, the detection area DAe includes a sentence “AAA ...”, the detection area DAf includes a sentence “BBB ...”, and the detection area DAg “CCC ...”. The detection area DAh includes a sentence written “DDD...”.

  In the initial state shown in FIG. 11A, the characters included in each detection area DA are displayed in the same typeface. On the other hand, in the state where the novel is read by the user shown in FIG. 11B, since the read process is executed, the typeface of the character in the detection area DA determined to have been read is changed to italic. In addition, the character size is reduced. Further, a ruled line is attached to the character in the detection area DA including the line-of-sight position EP, and the character size is increased. Then, the characters in the detection area DA that are not determined to have been read remain in the initial state.

  As described above, since the user can intuitively recognize the part that has been read, the part that is currently being read, and the part that has not yet been read, the convenience of reading with an electronic document is improved.

In addition, when a plurality of pages are included in an electronic document as in a novel, if it is determined that all of the displayed page contents have been read, a page change process is executed as a predetermined process. Is displayed. That is, the user can display the next page simply by reading the sentence. However, since the user may reconfirm the previous page, the user operates the first operation key 22a or the second operation key 22b to display the next page (or the previous page). Can do.

  Even when a novel electronic document is displayed, a reading process is executed as a predetermined process when the page has been read. Further, when the electronic document is a novel, the time is recorded as a reading log when the reading completion process is executed. And since the user can grasp | ascertain the time passage which read the novel by confirming the reading log, the convenience of the reading using an electronic document further improves.

  Note that the detection area DA in the novel may include lines, paragraphs, sections, chapters, and the like instead of sentences. When the electronic document terminal 10 includes a touch panel, the page update process may be executed according to a touch operation instead of operating the first operation key 22a or the second operation key 22b.

  In addition, when the already-read process is executed, not only the character typeface and character size but also the character color may be changed.

  Referring to FIGS. 12A and 12B, when a comic whose type is “document / image” is displayed, “frame” in the comic corresponds to detection area DA. The “frame” includes an image such as a character and a background, and a speech bubble F in which the character's dialogue is written. 12 displays a so-called “four-frame cartoon” composed of four frames. The detection area DAj, which is the first frame, includes a female character balloon Fa, and the detection area DAk, which is the second frame, includes a male character balloon Fb. The area DAm includes a female character balloon Fc, and the detection area DAn in the fourth frame includes a male character balloon Fd. However, in the initial state shown in FIG. 12A, no dialogue (sentence) is written in the balloon of each frame.

  In such a state, when the user's line-of-sight position EP is included in the first frame (detection area DAj) and the line-of-sight speed ES and the stay time ET satisfy the conditions based on the first threshold and the second threshold, A first additional information display process is executed.

Specifically, when the first additional information display process is executed, dialogue (“###”) is displayed as additional information for the balloon Fa. Similarly, when it is determined that the second frame (detection area DAk) has been read, the speech (“***”) is displayed in the balloon Fb. That is, by displaying additional information on a frame after it has been determined that the comic frame has been read, it is possible to assist in reading the comic.

  Referring to FIG. 12B, when the current line-of-sight position EP is included in the second frame (detection area DAk), the dialogue written in the balloon Fb is underlined. On the other hand, in the dialogue written in the balloon Fa of the first frame (detection area DAj) that has already been read, the character size is reduced and the typeface is changed to italic. That is, since the read process is executed even in the case of a comic, the user can intuitively recognize a frame that has been read, a frame that is currently being read, and a frame that has not been read, as in the case of a novel. Therefore, convenience when reading comics is improved.

As in the case of the novel, when it is determined that all the frames (detection area DA) have been read, the next page is displayed. In addition, when the electronic document terminal 10 includes a speaker, dialogue may be displayed in the balloon F, and an effect such as an additional concentration line, a color may be added to the image, or a sound effect may be output. Furthermore, when the electronic document terminal 10 includes a vibrator,
The dialogue may be displayed in the balloon F and vibrated.

Referring to FIGS. 13A and 13B, when a picture book whose electronic document type is “image” is displayed, in one page, a detection area DA in which a sentence and a picture (image) are different from each other is displayed. Is displayed. However, in the initial state, a character string “+++” is written in the detection area DAp, but nothing is drawn in the detection area ADq.

  In this state, if the line-of-sight speed ES and the stay time ET based on the user's line-of-sight position EP satisfy the conditions based on the first threshold and the second threshold, the second additional information display process is executed as a predetermined process. As a result, a picture (image) is displayed as additional information for the detection area ADq. As a result, the user who reads the picture book can be surprised by a visual change.

  Further, when it is determined that the line-of-sight position EP is moved to the detection area DAq where the image is displayed and the detection area DAq where the image is displayed is viewed (read), the next page is displayed after a certain time has passed. Is displayed. This is because if it is determined that the detection area DAq for displaying a picture (image) has been read and the next page is displayed immediately, the user cannot appreciate the picture (image) slowly. Therefore, in this embodiment, the user can appreciate the displayed picture by displaying the next page after a predetermined time has elapsed.

  When additional information such as an image is displayed, sound effects, music, and the like may be played back. Further, the additional information to be displayed may be a still image and a moving image. In another embodiment, it may be determined that a sentence (line) is read and at the same time, a related image (picture) is also read.

  As described above, in this embodiment, the user can read the electronic document without complicated operations only by browsing the electronic document.

  Moreover, since the electronic document terminal 10 can browse an electronic document without using a touch panel, the visibility of the display 26 is not reduced by a touch operation. For example, in the case of an electronic document terminal provided with a touch panel on the display 26, the page update process is executed by a touch operation, so that a fingerprint such as a finger performing the touch operation adheres to the touch panel as dirt. In this case, since the visibility of the display 26 is deteriorated due to the dirt, the user has to wipe off the dirt when continuously viewing the electronic document. However, in the electronic document terminal 10 of the present embodiment, since the page can be easily updated without performing a touch operation, the visibility of the display 26 is lowered even if the user continuously browses the electronic document. Never do.

  Further, since the electronic document can be read without using the hand, even the handicapped user can use the electronic document terminal 10 of the present embodiment.

  FIG. 14 is a diagram showing a memory map of the RAM 30. The RAM 30 includes a program storage area 302 and a data storage area 304. A part of the program and data is read from the flash memory 28 all at once or partially and sequentially as needed and stored in the RAM 30.

  A program storage area 302 stores a program for operating the electronic document terminal 10. For example, a program for operating the electronic document terminal 10 includes an electronic document control program 310, a text display processing program 312, a text / image display program 314, an image display program 316, a line-of-sight position detection program 318, and the like.

  The electronic document control program 310 is a program that is executed to view an electronic document. The text display program 312 is a program for executing predetermined processing based on the detected line-of-sight position EP in an electronic document of the type “text” such as a contract or a novel. Similarly, the sentence / image display program 314 is a program for executing a predetermined process in an electronic document of the type “sentence / image” such as a comic. The image display program 316 is a program for executing predetermined processing on an electronic document whose type is “image”, such as a picture book. The line-of-sight position detection program 318 is a program for detecting the line-of-sight position EP of the user, calculating the line-of-sight speed ES and the staying time ET, and updating the state table.

  Although illustration is omitted, the program for operating the electronic document terminal 10 includes a program corresponding to the basic software (OS) of the electronic document terminal.

  Subsequently, the data storage area 304 stores a state table buffer 330, electronic document file data 332, and the like. The status table buffer 330 temporarily stores the status table shown in FIG.

  The electronic document file data 332 is data that is read from the flash memory 28 and temporarily stored in the RAM 30 when browsing the electronic document. The electronic document file data 332 includes metadata 332a, document data 332b, interruption data 332c, and the like. The metadata 332a is data having a configuration shown in FIG. 7B, for example. The document data 332b is data read when an electronic document is displayed, and includes text data and image data. The interruption data 332c is data that is created and stored when the electronic document is interrupted before reading to the last page. Therefore, in the case of electronic document file data 332 for which browsing has not been interrupted, this interruption data 332c is not included.

  Although illustration is omitted, the data storage area 304 stores image data displayed in a standby state, character string data, and the like, and includes counters and flags necessary for the operation of the electronic document terminal 10. Provided.

  The processor 20 controls the electronic document control process shown in FIG. 15, the document display process shown in FIG. 16, the document / image display process shown in FIG. A plurality of tasks are processed in parallel, including the image display process shown in FIG. 18 and the line-of-sight position detection process shown in FIG.

  For example, when an operation for browsing an electronic document is performed, the processor 20 reads the metadata 332a included in the electronic document file data 332 in step S1. That is, by reading the metadata 332a, the type of electronic document, the coordinate range of the detection area DA, the first threshold value, the second threshold value, the order, the start area, and the like are acquired. Subsequently, in step S3, page display processing is executed. That is, the first page in the electronic document is displayed on the display 26. However, when the interruption data 332c is included in the electronic document file data 332, it is displayed on the display 26 recorded in the interruption data 332c. If “NO” is determined in step S23 described later and the process in step S3 is executed, the next page is displayed on the display 26. Therefore, the processor 20 that executes the process of step S3 functions as a next page display unit or an execution unit.

Subsequently, in step S5, it is determined whether or not eye gaze position detection is necessary. For example, it is determined whether it is necessary to detect the line-of-sight position based on whether the first threshold value or the second threshold value is stored in the read metadata 332a. If “NO” in the step S5, for example, if the metadata 332a does not include the first threshold value or the second threshold value, it is not necessary to detect the eye-gaze position. Proceed to S23. That is, an instruction to end the line-of-sight position detection process shown in FIG. 19 is issued in step S7. However, if the line-of-sight position detection process is not executed, the issued end command is invalid.

  On the other hand, if “YES” in the step S5, for example, if one of the threshold values is included in the metadata 332a, the line-of-sight position detection process is executed in a step S9. That is, an instruction to execute the line-of-sight position detection process is issued in step S9. Subsequently, in step S11, it is determined whether or not the line-of-sight position EP is included in the start area. That is, it is determined whether the coordinates of the line-of-sight position EP in the state table are included in the coordinate range of the start area included in the metadata 332a. If “NO” in the step S11, for example, if the coordinates of the line-of-sight position EP are included in the coordinate range of the start area, the process of the step S11 is repeated.

  If “YES” in the step S11, for example, referring to FIG. 7B and FIG. 10, the coordinates (Ex, Ey) of the line-of-sight position EP of the state table are the coordinates of the detection area DAa which is the start area. If it is not included in the range ((X1, Y1)-(X2, Y2)), it is determined in step S13 whether the type of the electronic document is a sentence. That is, it is determined whether the type included in the metadata 332a is “text”. If “YES” in the step S13, that is, if the type of the electronic document is a sentence, a document display process is executed in a step S15, and the process proceeds to a step S23. If “NO” in the step S13, that is, if the electronic document type is not “sentence”, it is determined whether or not the electronic document type is a sentence / image in a step S17. If “YES” in the step S17, that is, if the type of the electronic document is “sentence / image”, a sentence / image display process is executed in a step S19, and the process proceeds to a step S23. If “NO” in the step S17, that is, if the type of the electronic document is not “text / image”, the type is “image”, and therefore the image display process is executed in a step S21, and the process proceeds to the step S23. .

  Note that the sentence display process in step S15, the sentence / image display process in step S19, and the image display process in step S21 will be described with reference to the flowcharts of FIGS. 16 to 18, and detailed description thereof is omitted here. To do.

  In step S23, it is determined whether it is the last page. That is, it is determined whether the number of pages in the state table has reached the last page. If “NO” in the step S23, that is, if the current page number has not reached the final page, the process returns to the step S3. On the other hand, if “YES” in the step S23, that is, if the current page number has reached the last page, the electronic document control process is ended.

  FIG. 16 is a flowchart of the document display process. When step S15 is executed in the electronic document control process shown in FIG. 15, the processor 20 acquires the line-of-sight position EP in step S41. That is, the coordinates of the line-of-sight position EP are read from the state table. Subsequently, in step S43, it is determined whether or not the line-of-sight position EP is outside the display range. That is, it is determined whether or not the user is looking at the display 26. If “YES” in the step S43, that is, if the detected line-of-sight position EP is outside the display range of the display 26, the backlight is turned off in a step S45, and the process proceeds to the step S59. For example, since the user is not looking at the display 26, the power of the backlight is turned off to reduce power consumption. If “NO” in the step S43, that is, the user is looking at the display 26, the backlight is turned on in a step S47. That is, the backlight is turned on. Note that when the backlight is already lit (turned off), the luminance of the backlight does not change.

  Subsequently, in step S49, it is determined whether or not the line-of-sight position EP is included in the detection area DA. That is, the detection area DA to which the user's line of sight should be directed is selected according to the order stored in the metadata 332a, and it is determined whether the coordinate range of the detection area DA includes the coordinates of the line-of-sight position EP. If “NO” in the step S49, that is, if the coordinate range of the line-of-sight position EP is not included in the coordinate range of the detection area DA selected as described above, the process proceeds to a step S59.

If “YES” in the step S49, that is, if the coordinates of the visual line position EP are included in the coordinate range of the detection area DA, it is determined whether or not the visual line speed ES is equal to or less than the first threshold value in a step S51. The That is, it is determined whether or not the line-of-sight velocity ES _n recorded in the state table is equal to or lower than the first threshold value stored in the metadata 332a. If “NO” in the step S51, for example, if the current line-of-sight speed ES _n is faster than THa [mm / ms], the step S59 is performed.
Proceed to

On the other hand, if "YES" in the step S51, for example, the current radial velocity ES _n equal to or less than THa [mm / ms], the staying time ET in step S53, it is determined whether the second threshold value or more. That is, it is determined whether or not the current stay time ET _n stored in the state table is _equal to or greater than the second threshold included in the metadata 332a. If “NO” in the step S53, for example, if the stay time ET _n in the detection area DAa is shorter than THb [ms], the process proceeds to a step S59. On the other hand, if “YES” in the step S53, for example, if the stay time ET _n in the detection area DAa is equal to or longer than THb [ms], the display state of the detection area DA including the line-of-sight position EP is changed in a step S55. . For example, when the process of step S55 is executed, as shown in FIG. 8B, the typeface of the character included in the detection area DAa is italicized and the character size is reduced.

  Subsequently, in step S57, it is determined whether or not it is the last detection area DA. That is, it is determined whether the current line-of-sight position EP recorded in the state table is included in the coordinate range of the last detection area DA in the order stored in the metadata 332a. If “NO” in the step S57, that is, if it is not the last detection area DA, it is determined whether or not an end operation is performed in a step S59. For example, it is determined whether or not the function key 22c is pressed for a long time. If “NO” in the step S59, that is, if the end operation is not performed, the process returns to the step S41. On the other hand, if “YES” in the step S59, that is, if an end operation is performed, the interruption data 332c is stored in a step S61. That is, the interruption data 332c created based on the state table is stored in the RAM 30. When the process of step S61 ends, the document display process ends, and the process returns to the electronic document control process. The processor 20 that executes the process of step S59 functions as an end operation determination unit, and the processor 20 that executes the process of step S61 functions as a storage unit.

  If “YES” in the step S57, for example, as shown in FIG. 9, when the display state of all the detection areas DAd is changed, a reading process is executed in a step S63. For example, when the electronic document is a contract, when the reading process is executed, the agreement key 70 and the disagreement key 72 are displayed in a selectable manner as shown in FIG. When the electronic document is a novel, when the reading completion process is executed, the time at that time is stored as a reading log. When the reading completion process is executed, the document display process is terminated and the process returns to the electronic document control process.

  Note that the processor 20 that executes the processing of steps S49-53 functions as a determination unit. The processor 20 that executes the processes of steps S55 and S63 functions as an execution unit. And the processor 20 which performs the process of step S63 functions as a key display part.

  FIG. 17 is a flowchart of the text / image display process. However, in the sentence / image display process, the same step number is assigned to the same process as the document display process, and detailed description of the step is omitted.

  When step S19 is executed in the electronic document control process shown in FIG. 15, the processor 20 acquires the line-of-sight position EP in step S41, and determines whether the line-of-sight position EP is outside the display range of the display 26 in step S43. To do. If “YES” in the step S43, that is, if the line-of-sight position EP is out of the display range, the backlight is erased in a step S45, and the process proceeds to the step S59. On the other hand, if “NO” in the step S43, that is, if the line-of-sight position EP is within the display range, the backlight is turned on in a step S47.

  Subsequently, in step S49, it is determined whether or not the line-of-sight position EP is included in the detection area (cartoon frame) DA. If “YES” in the step S49, that is, if the detected line-of-sight position EP is included in the detection area DA, it is determined whether or not the line-of-sight speed ES is equal to or less than the first threshold value in a step S51. If “YES” in the step S51, that is, if the calculated line-of-sight speed ES is equal to or smaller than the first threshold value, it is determined whether or not the stay time ET is equal to or larger than the second threshold value in a step S53. If the stay time ET calculated in step S51 is greater than or equal to the second threshold, the process proceeds to step S71. If “NO” in the steps S49, S51, S53, the process proceeds to a step S59.

  In step S71, additional information is displayed in the detection area DA. For example, as shown in FIG. 12B, the dialogue is displayed in the balloon F in the frame (detection area DA). Note that the processor 20 that executes the process of step S71 functions as a first additional information display unit.

  Subsequently, in step S57, it is determined whether or not it is the last detection area (frame) DA. If “NO” in the step S57, that is, if the detection area DA determined to be read is not the last detection area DA, it is determined whether or not an end operation is performed in a step S59. If “NO” in the step S59, that is, if the end operation is not performed, the process returns to the step S41. On the other hand, if “YES” in the step S59, that is, if an ending operation is performed, the interruption data is stored in a step S61, the sentence / image display process is ended, and the process returns to the electronic document control process. Even if “NO” in step S57, that is, even if the detection area DA determined to have been read is the last detection area DA, the sentence / image display process is terminated.

  FIG. 18 is a flowchart of image display processing. However, in this process, the same step number is attached | subjected to the step same as a text display process, and detailed description is abbreviate | omitted about the step.

  When step S21 is executed in the electronic document control process shown in FIG. 15, the processor 20 acquires the line-of-sight position EP in step S41, and determines in step S43 whether the line-of-sight position EP is outside the display range. If “YES” in the step S43, that is, if the line-of-sight position EP is out of the display range, the backlight is turned off in a step S45, and the process proceeds to the step S59. On the other hand, if “NO” in the step S43, that is, if the user's line-of-sight position EP is not outside the display range, the backlight is turned on in a step S47.

Subsequently, in step S49, it is determined whether or not the line-of-sight position EP is included in the detection area DA. If “YES” in the step S49, that is, if the line-of-sight position EP is included in the detection area DA, it is determined whether or not the line-of-sight speed ES is equal to or less than the first threshold value in a step S51. If “YES” in the step S51, that is, if the line-of-sight speed ES is not more than the first threshold value, it is determined whether or not the stay time ET is on the second threshold value in a step S53. If “YES” in the step S53, that is, if the stay time ET is equal to or longer than the second threshold value, the process proceeds to a step S81.

  In step S81, additional information is displayed in another detection area DA. For example, as shown in FIG. 13B, a picture (image) is displayed in the detection area DAq corresponding to the detection area DAp determined to have been read. Subsequently, in step S83, standby processing is executed. That is, it waits for a certain period of time so that the additionally displayed image can be viewed. Then, when a certain time has elapsed, the image display process is terminated and the process returns to the electronic document control process. The processor 20 that executes the process of step S81 functions as a second additional information display unit.

  If “NO” in the steps S49, S51, and S53, it is determined whether or not an end operation has been performed in a step S59. If “NO” in the step S59, that is, if the end operation is not performed, the process returns to the step S41. If “YES” in the step S59, that is, if an ending operation is performed, the interruption data is stored in a step S61, and the image display unit is ended.

  FIG. 19 is a flowchart of the gaze position detection process. For example, when step S9 of the electronic document control process is executed and an execution instruction for the line-of-sight position detection process is issued, the processor 20 acquires a captured image at step S91. That is, the captured image stored in the RAM 34 is read out. Subsequently, in step S93, it is determined whether or not the captured image includes a face area. That is, image processing for extracting a face area is added to the photographed image.

  If “YES” in the step S93, for example, if the photographed image includes the user's face area, the eye area is extracted in a step S95. That is, image processing for extracting a region having the eye feature amount is added to the photographed image. Subsequently, in step S97, the line-of-sight direction is estimated. That is, in step S97, the user's line-of-sight direction is estimated by calculating the center of the black eye area based on the extracted eye area. Subsequently, in step S99, the line-of-sight position EP is detected from the estimated line-of-sight direction. That is, the line-of-sight position EP is detected based on the line-of-sight direction and the line-of-sight distance between the user's face and the display 26. The processor 20 that executes the process of step S99 functions as a line-of-sight position detection unit.

Subsequently, in step S101, the line-of-sight speed ES is calculated. That frame to calculate a time Delta] t [ms] and the instantaneous speed s _{m [mm} / ms] for each frame based on the change amount d [mm], further, the instantaneous velocity s _m at a given time _[mm / ms] The average is calculated as the line-of-sight speed ES. The processor 24 that executes the process of step S101 functions as a line-of-sight speed calculation unit.

  Subsequently, in step S103, it is determined whether or not the line-of-sight position EP is included in the detection area DA. For example, it is determined whether or not the detected line-of-sight position EP is included in any one of the detection areas DA included in the displayed electronic document. If “NO” in the step S103, that is, if the detected eye-gaze position EP is not included in the detection area DA, the process proceeds to a step S107. On the other hand, if “YES” in the step S103, as shown in FIG. 8B, for example, if the detected line-of-sight position EPm is included in the detection area DAb, the stay time ET is measured in a step S105. Specifically, first, it is determined whether the line-of-sight position EP detected in the current process is included in the same detection area DA as the line-of-sight position EP detected in the previous process. Next, if they are included in the same detection area DA, they are counted, and the stay time ET is obtained based on the frame time Δt and the counted number. In addition, the processor 20 which performs the process of step S105 functions as a stay time measurement part.

In step S107, the state table is updated. For example, as shown in FIG. 10, the detected line-of-sight position E, line-of-sight speed ES, and stay time ET are recorded in the state table. And
When the process of step S107 is completed, the process returns to step S91.

  If “NO” in the step S93, that is, if the captured image does not include a face area, it is determined whether or not an end command is issued in a step S109. If “NO” in the step S109, that is, if an end instruction is not issued, the process returns to the step S91. On the other hand, if “YES” in the step S109, for example, a step S7 of the electronic document control process is executed, and when a gaze position detection process end command is issued, the gaze position detection process ends.

  Note that a pointer such as an arrow may be displayed near the line-of-sight position EP so that the user can grasp the detected line-of-sight position EP.

  Further, when the detection area DA is surrounded by a frame such as a cartoon, the frame of the detection area DA including the line-of-sight position EP is different from the frame of the other detection areas DA in terms of color, line width, and shape. Different display modes may be used. In addition, the color of the frame of the detection area DA including the line-of-sight position EP may be red when additional information exists, and may be black when no additional information exists.

  In the metadata 332a, a plurality of detection areas DA may be set as start areas. The first threshold value and the second threshold value may be set for each page of the electronic document. Further, only one of the first threshold value and the second threshold value may be recorded in the metadata 332a. Further, the start area, the order, and the like may not be displayed in the metadata. That is, how to read an electronic document is not necessarily limited by the creator of the electronic document. Each detection area DA may be set manually by the creator of the electronic document, or may be set automatically using an image recognition process. Furthermore, the first threshold value and the second threshold value may be different depending on the type of electronic document.

  In addition, correction processing (calibration) may be performed every time an electronic document is displayed in order to improve the accuracy of eye gaze position detection. For example, the accuracy of detection of the line-of-sight position EP is improved by causing the user to pay attention to the four corners of the display 26 in order and correcting the error from the line-of-sight position EP at that time.

  In another embodiment, if the line-of-sight speed ES is faster than a specified value, the mode may be shifted to the fast reading mode. When the mode is changed to the fast reading mode, words representing contents such as nouns and verbs are highlighted with an underline. On the other hand, characters that connect sentences such as other articles, prepositions, pronouns, and the like are displayed with a reduced character size.

  Further, the electronic document terminal 10 may be able to easily add electronic document file data by enabling communication with an external terminal or a network.

  The plurality of programs used in the present embodiment may be stored in the HDD of the data distribution server and distributed to the electronic document terminal 10 via the network. Further, the storage medium may be sold or distributed in a state where a plurality of programs are stored in a storage medium such as an optical disk such as a CD, DVD, or BD (Blu-ray Disc), a USB memory, or a memory card. When a plurality of programs downloaded through the above-described server or storage medium are installed in an electronic document terminal having the same configuration as that of this embodiment, the same effect as that of this embodiment can be obtained.

  Furthermore, this embodiment may be applied not only to the electronic document terminal 10 but also to a so-called smart phone, PDA (Personal Digital Assistant), desktop PC, and notebook PC.

DESCRIPTION OF SYMBOLS 10 ... Electronic document terminal 20 ... Processor 22 ... Key input device 26 ... Display 30 ... RAM
36 ... Camera control circuit 38 ... Image sensor 40 ... Focus lens OP1 ... Aperture

1st invention is the electronic device which has a display part which displays an electronic document, Comprising: The imaging | photography part which image | photographs an image, The gaze position which detects the gaze position with respect to a display part based on the image image | photographed by the imaging | photography part When the determination unit determines whether the electronic document displayed on the display unit is read based on the line-of-sight position detected by the detection unit, the line-of-sight position detection unit, and when the electronic document is read by the determination unit, An execution unit that executes predetermined processing related to an electronic document is displayed , and the execution unit displays characters included in a detection area that is determined to be read in the electronic document and characters included in a detection area that is not determined to be read. It is an electronic device which makes an aspect different .

In the first invention, the display unit (26) of the electronic device (10: reference numerals exemplifying corresponding parts in the embodiment, hereinafter the same) is, for example, an LCD monitor, and is a contract, novel, comic book and picture book. Display an electronic document. The imaging unit (36-40) captures an image of the user, for example, and the line-of-sight position detection unit (20, S99) detects the user's line-of-sight position (EP) based on the image. The determination unit (20, S49-S53) determines whether the displayed electronic document has been read based on the detected movement of the line-of-sight position. When it is determined that the electronic document has been read, the execution unit (20, S3, S55, S63, S71, S81) executes a predetermined process of displaying the next page or displaying additional information. Specifically, the execution unit changes the display mode between the character included in the detection area determined to be read and the character included in the detection area not determined to be read. For example, the former character is changed to italic and the character size is reduced, and the latter character is left in the initial state.

According to the first invention, the user can read the electronic document without performing complicated operations only by browsing the electronic document. In particular, since the user can intuitively recognize the part that has been read and the part that has not yet been read, the convenience of reading with an electronic document is improved.
A second invention is dependent on the first invention, and the execution unit has different display modes between the character included in the detection area determined to be read in the electronic document and the character in the detection area including the line-of-sight position. Electronic equipment.
In the second invention, the execution unit, for example, changes the typeface of the character in the detection area determined to have been read to an italic type and reduces the character size, but adds a ruled line to the character in the detection area including the line-of-sight position. And increase the character size.
According to the second aspect of the invention, the user can intuitively recognize the part that has been read and the part that is currently being read, thereby improving the convenience of reading with an electronic document.

A third invention is dependent on the first or second invention, and further includes a line-of-sight speed calculation unit that calculates a movement speed of the line of sight based on the line-of-sight position detected by the line-of-sight position detection unit, The electronic device determines that the electronic document displayed on the display unit has been read when the line-of-sight velocity calculated by the line-of-sight velocity calculation unit is equal to or less than the first threshold value.

In the third invention, the line-of-sight speed calculation unit (20, S101) calculates the line-of-sight speed (ES) every predetermined time based on the change in the line-of-sight position for each frame time. If the line-of-sight speed at the line-of-sight position moves below the first threshold (THa), it is determined that the displayed electronic document has been read.

According to the third aspect , it is possible to determine whether the content of the electronic document has been read using the line-of-sight speed based on the line-of-sight position.

A fourth invention is dependent on any one of the first to third inventions, and the electronic document includes a detection area for displaying a sentence or an image, and the line-of-sight position detected by the line-of-sight position detection unit is the detection area. When the stay time measured by the stay time measuring unit is more than a second threshold, the electronic document displayed on the display unit is read when the stay time measuring unit that measures the staying time is further included. Judgment is an electronic device .

In the fourth invention, the electronic document includes a detection area (DA) in which a sentence or an image is displayed. The stay time measuring unit (20, S105) measures the stay time (ET) in which the detected line-of-sight position stays in the detection area. If the stay time of the line-of-sight position in the detection area is equal to or greater than the second threshold (THb), it is determined that the displayed electronic document has been read.

According to the fourth invention, it is possible to determine whether the content of the electronic document has been read by using the stay time based on the line-of-sight position.

5th invention has the display part (26) which displays an electronic document, and the imaging | photography part (36-40) which image | photographs an image which was image | photographed by the imaging | photography part about the processor (20) of the electronic device (10). The line-of-sight position detection unit (S99) for detecting the line-of-sight position relative to the display unit, and the determination for determining whether the electronic document displayed on the display unit has been read based on the line-of-sight position detected by the line-of-sight position detection unit When the electronic document is read by the units (S49-S53) and the determination unit, the electronic document is caused to function as an execution unit (S3, S55, S63, S71, S81) that executes a predetermined process related to the electronic document. a character contained in the detection region which is determined to have been read, to vary the display mode between the characters included in the detection area which is not judged to have been read, electronic document control programming It is a non.

In the fifth invention, the same effect as in the first invention can be expected.

An eleventh invention is a control method for an electronic document equipment (10) electronic, a display step of displaying an electronic document, comprising the steps of capturing an image, based on the captured image, line-of-sight for the electronic document detecting the position, on the basis of the detected viewpoint position, a step (S49-S53) for determining whether the electronic document has been read, when the electronic document is determined to have been read, a predetermined processing related to the electronic document And an execution step (S3, S55, S63, S71, S81) . The execution step reads the characters included in the detection area determined to have been read in the electronic document. An electronic document control method including changing a display mode according to a character included in a detection area that is not determined to have been detected.

In the sixth invention, the same effect as in the first invention can be expected.

Furthermore, this embodiment may be applied not only to the electronic document terminal 10 but also to a so-called smart phone, PDA (Personal Digital Assistant), desktop PC, and notebook PC.
An embodiment is an electronic device having a display unit that displays an electronic document, and includes a photographing unit that captures an image, and a gaze position detecting unit that detects a gaze position with respect to the display unit based on an image captured by the photographing unit. A determination unit that determines whether the electronic document displayed on the display unit has been read based on the line-of-sight position detected by the line-of-sight position detection unit, and the electronic document when the determination unit determines that the electronic document has been read. It is an electronic device provided with the execution part which performs the predetermined process regarding.
In this embodiment, the display unit (26) of the electronic device (10: reference numerals exemplifying corresponding parts in the embodiment; hereinafter the same) is, for example, an LCD monitor, such as a contract, novel, comic book, picture book, and the like. Display electronic documents. The imaging unit (36-40) captures an image of the user, for example, and the line-of-sight position detection unit (20, S99) detects the user's line-of-sight position (EP) based on the image. The determination unit (20, S49-S53) determines whether the displayed electronic document has been read based on the detected movement of the line-of-sight position. When it is determined that the electronic document has been read, the execution unit (20, S3, S55, S63, S71, S81) executes a predetermined process of displaying the next page or displaying additional information.
According to this embodiment, the user can read the electronic document without complicated operations only by browsing the electronic document.
Another embodiment is dependent on the above-described embodiment, and the electronic document includes a detection area for displaying a sentence or an image, and the line of sight calculates the movement speed of the line of sight based on the line-of-sight position detected by the line-of-sight position detection unit. The gaze position detected by the speed calculation unit and the gaze position detection unit further includes a stay time measurement unit that measures the time during which the gaze position stayed in the detection region, and the determination unit is the gaze speed calculated by the gaze speed calculation unit Is less than or equal to the first threshold value and the stay time measured by the stay time measuring unit is greater than or equal to the second threshold value, it is determined that the electronic document displayed on the display unit has been read.
In another embodiment, the electronic document includes a detection area (DA) where text or images are displayed. The line-of-sight speed calculation unit (20, S101) calculates the line-of-sight speed (ES) every predetermined time based on the change in the line-of-sight position for each frame time. The stay time measuring unit (20, S105) measures the stay time (ET) in which the detected line-of-sight position stays in the detection area. Then, if the line-of-sight speed of the line-of-sight position moves below the first threshold (THa) and the staying time of the line-of-sight position in the detection area is above the second threshold (THb), it is determined that the displayed electronic document has been read. The
According to another embodiment, it is possible to determine whether the content of the electronic document has been read using the line-of-sight speed and the staying time based on the line-of-sight position.
Another embodiment is dependent on any of the above embodiments, and an end operation determination unit that determines whether or not the end operation has been performed, and when the end operation determination unit determines that the end operation has been performed, the line-of-sight position detection unit Is further provided with a storage unit that stores interruption data based on the line-of-sight position detected by the above.
In another embodiment, the end operation determination unit (20, S59) determines whether an end operation for finishing the display of the electronic document has been performed. When the display of the electronic document is finished, the storage unit (20, S61) stores the detection area including the line-of-sight position, the page number to be displayed, and the like as interruption data.
According to another embodiment, even if the user interrupts the browsing of the electronic document, the reading of the electronic document can be easily resumed by reading the interruption data.
Still another embodiment is dependent on any of the above embodiments, and the electronic document includes a plurality of pages, and when the execution unit determines that one page of the electronic document has been read, the execution page displays the next page. It further includes a next page display section to be displayed.
In yet another embodiment, the electronic document includes a plurality of pages. For example, when it is determined that a certain page has been read, the next page display unit (20, S3) displays the next page.
According to another embodiment, the user can display the next page only by reading the sentence.
Another embodiment is dependent on any of the above embodiments, and the execution unit is configured to agree or disagree with the content of the electronic document when the determination unit determines that the electronic document has been read. A key display unit that displays keys to be operable is included.
In another alternative embodiment, the electronic document includes a contract, for example. If it is determined that the contract has been read, the key display unit (20, S63) displays a key for accepting or disagreeing with the contents of the contract in an operable manner. The user uses these keys to agree or disagree with the contents of the contract.
According to another embodiment, it is possible to prevent the user from agreeing without grasping the contents of the electronic document or from consenting by an erroneous operation.
Still another embodiment is dependent on any one of the above-described embodiments, and when the execution unit determines that the detection region of the electronic document has been read by the determination unit, the execution unit adds additional information to the detection region. A first additional information display unit for displaying is further included.
In yet another embodiment, for example, the electronic document includes a comic and the detection area includes a comic frame. Moreover, a 1st additional information display part (20, S71) displays a dialog as additional information in the balloon (F) contained in a comic frame, when the detection area | region containing a comic frame is read.
According to still another embodiment, it is possible to assist reading of the electronic document by displaying additional information after it is determined that a part of the electronic document has been read.
The other embodiments are dependent on any of the above-described embodiments, and the execution unit displays the text when the determination unit determines that the detection area where the text included in the electronic document is displayed has been read. And a second additional information display unit for displaying additional information in a detection area different from the detection area being used.
In other embodiments, the electronic document includes, for example, a picture book. For example, when the second additional information display unit (20, S81) determines that the detection area in which the dialogue is written has been read, an image (picture) corresponding to the dialogue is displayed in the other detection areas as additional information. The
According to another embodiment, a user who reads an electronic document can be surprised by a visual change.

Claims (11)

An electronic device having a display unit for displaying an electronic document,
Shooting unit for shooting images,
A line-of-sight position detection unit for detecting a line-of-sight position with respect to the display unit based on an image photographed by the photographing unit;
Based on the line-of-sight position detected by the line-of-sight position detection unit, a determination unit that determines whether the electronic document displayed on the display unit has been read, and when the determination unit determines that the electronic document has been read An electronic device comprising an execution unit that executes predetermined processing relating to an electronic document. Based on the line-of-sight position detected by the line-of-sight position detection unit, further comprises a line-of-sight speed calculation unit that calculates the movement speed of the line of sight,
The electronic device according to claim 1, wherein the determination unit determines that the electronic document displayed on the display unit has been read when the line-of-sight velocity calculated by the line-of-sight velocity calculation unit is equal to or less than a first threshold. The electronic document includes a detection area for displaying a sentence or an image,
The eye gaze position detected by the eye gaze position detecting unit further includes a stay time measuring unit that measures the time during which the eye gaze position has stayed in the detection region,
The electronic device according to claim 1, wherein the determination unit determines that the electronic document displayed on the display unit has been read when the stay time measured by the stay time measurement unit is equal to or greater than a second threshold. The electronic document includes a detection area for displaying a sentence or an image,
Based on the line-of-sight position detected by the line-of-sight position detection unit, the line-of-sight speed calculation unit that calculates the movement speed of the line of sight, and the time during which the line-of-sight position detected by the line-of-sight position detection unit stayed in the detection region It further includes a stay time measurement unit that measures
The determination unit is an electronic device that is displayed on the display unit when the line-of-sight velocity calculated by the line-of-sight velocity calculation unit is equal to or less than a first threshold value and the stay time measured by the stay time measurement unit is equal to or greater than a second threshold value. The electronic device according to claim 1, wherein the electronic device determines that the document has been read. An end operation determining unit that determines whether an end operation has been performed, and a memory that stores interruption data based on the line-of-sight position detected by the line-of-sight position detecting unit when the end operation determining unit determines that the end operation has been performed. The electronic device according to claim 1, further comprising a unit. The electronic document includes a plurality of pages,
The electronic device according to claim 1, further comprising a next page display unit that displays a next page when it is determined that one page of the electronic document has been read.   The execution unit includes a key display unit that operably displays a key for agreeing or disagreeing with the contents of the electronic document when the determining unit determines that the electronic document has been read. The electronic device in any one of 6 thru | or 6.   The said execution part further contains the 1st additional information display part which displays additional information on the detection area, when it is judged that the detection area of the said electronic document was read by the said judgment part. The electronic device according to Crab. The execution unit displays additional information in a detection region different from the detection region where the sentence is displayed when the determination unit determines that the detection region where the sentence included in the electronic document is displayed has been read. The electronic device according to claim 3, further comprising a second additional information display unit. A processor of an electronic device having a display unit for displaying an electronic document and a photographing unit for photographing an image;
A line-of-sight position detection unit for detecting a line-of-sight position with respect to the display unit based on an image photographed by the photographing unit;
Based on the line-of-sight position detected by the line-of-sight position detection unit, a determination unit that determines whether the electronic document displayed on the display unit has been read, and when the determination unit determines that the electronic document has been read An electronic document control program that functions as an execution unit that executes predetermined processing relating to an electronic document. An electronic document control method for an electronic device having a display unit for displaying an electronic document and a photographing unit for photographing an image,
Based on the image photographed by the photographing unit, a line-of-sight position with respect to the display unit is detected,
An electronic document that determines whether an electronic document displayed on the display unit has been read based on the detected line-of-sight position, and executes predetermined processing on the electronic document when it is determined that the electronic document has been read Control method.

Images