JP2011243141A - Operation information processor, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve a problem that a device may execute an operation which does not correctly reflect a user's intention.SOLUTION: The operation information processor acquires distinction information to clarify whether a text, which is a display object displayed on a display means, is written vertically or horizontally; inputs images photographed by imaging means for photographing a face or eyes of a user who exists at a position where the user can view the text; detects movements of the user's face or eyes from the images; and is provided with detection means which extracts lateral movements out of the detected movements when the distinction information indicates vertical writing, and extracts and outputs longitudinal movements out of the detected movements when the distinction information indicates horizontal writing, and processing means for transmitting a prescribed operation instruction defined by the movements which are output by the detection means, to operation means for executing the operation instruction.

Description

  The present invention relates to an operation information processing apparatus, method, and program.

  A pointing device such as a mouse is used as an input device of a computer such as a PC (Personal computer) mainly for operating a pointer or an icon displayed on a screen.

  In addition, the pointing device requests an operation such as moving the pointer or pressing a button in a GUI (Graphical User Interface) environment that provides an intuitive input operation for a pointer, icon, button, or the like displayed on the display. It is used.

  However, since the pointing device needs to be directly operated by the user using his / her hand, there is a problem that it is inconvenient for an operator who has difficulty in hand operation due to physical injury or injury.

  An example of a technique for solving such a problem is described in Patent Document 1. A moving body detection apparatus that realizes a pointing device described in Patent Document 1 includes a receiving unit that receives a physical quantity corresponding to the movement of a moving body in time series, and a physical quantity received by the receiving unit over a predetermined time in coordinates in a predetermined coordinate system. Conversion means for converting into a plurality of coordinates, means for detecting the movement of the moving body based on fluctuations in the coordinates converted by the conversion means, means for storing a plurality of coordinates converted by the conversion means, A calculation unit that calculates a space on the coordinate system in which coordinates are included, and a coordinate that is obtained by converting a physical quantity received by the reception unit after the space is calculated is determined to be included in the space; And means for determining that the moving object is stopped when it is determined that the moving object is included in the space.

  Another example is described in Patent Document 2. The pointer control signal generation method described in Patent Document 2 includes a pointer position control interface displayed on a display by a signal from a pointing device and an interface for transmitting an operation instruction by a user push button associated with the pointer position to software. A method of generating a control signal for a system having a step of detecting movement of the head and generating movement amount information, a step of detecting opening and closing of the left eye and the right eye and generating opening and closing information, The step includes associating eye opening / closing information with the pressing state information of the left and right push button switches of the pointing device and outputting the movement amount information and the pressing state information as control signals.

Patent No. 4411387 JP-A-2005-352580

  The technology described in the above-described patent document determines the operation content related to the pointing device or the like only by the movement of the user's face or eyes, regardless of the display object such as the content displayed on the display screen. Therefore, there is a problem that the apparatus executes an operation that does not correctly reflect the user's intention.

  Therefore, an object of the present invention is to provide an operation information processing apparatus, method, and program for executing an operation that correctly reflects the user's intention.

  In order to achieve such an object, an embodiment of the present invention is an operation information processing apparatus that obtains distinction information about whether a sentence, which is a display object displayed on a display means, is vertical writing or horizontal writing, and uses the sentence as a user. The image information input by the imaging means for capturing the user's face or eyes existing at a position where the user can see the image, and the detection of the user's face or eye movement from the image, and the discrimination information When the vertical direction is written, the horizontal movement of the movement is extracted. When the discrimination information is horizontal writing, the vertical movement of the movement is extracted and output, and the detection section outputs the movement. And processing means for transmitting a predetermined operation instruction determined by movement to an operation means for executing the operation instruction.

  In addition, according to the present invention, it is possible to acquire the discrimination information whether the text that is the display object displayed on the display means is vertical writing or horizontal writing, and the user's face or the user existing at a position where the user can see the text or The image input picked up by the imaging means for picking up the eyes and the detection of the user's face or eye movement from the image are performed, and when the distinction information is vertical writing, the horizontal movement of the movement is performed. When the distinction information is horizontal writing, a detection step of extracting and outputting a vertical movement of the movement, and a predetermined operation instruction determined by the movement output by the detection means, the operation instruction There is provided an operation information processing program for causing a computer to execute processing steps to be transmitted to an operating means for executing.

  In addition, according to the present invention, it is possible to acquire the discrimination information whether the text that is the display object displayed on the display means is vertical writing or horizontal writing, and the user's face or the user existing at a position where the user can see the text or The image input picked up by the imaging means for picking up the eyes and the detection of the user's face or eye movement from the image are performed, and when the distinction information is vertical writing, the horizontal movement of the movement is performed. When the distinction information is horizontal writing, a detection step of extracting and outputting a vertical movement of the movement, and a predetermined operation instruction determined by the movement output by the detection means, the operation instruction And an operation information processing method for performing processing steps to be transmitted to the operating means for executing.

  According to the present invention, an operation information processing apparatus, method, and program for executing an operation that correctly reflects the user's intention are provided.

It is the schematic for demonstrating the outline | summary of 1st embodiment. It is a figure which shows an example of the block diagram of the operation information processing apparatus 80 of 1st embodiment. It is a figure for demonstrating the function of the detection control part 100 of 1st embodiment. It is a figure which shows an example of the operation condition memory | storage part 30 of 1st embodiment. It is a figure which shows an example of the operation condition memory | storage part 30 of 1st embodiment. It is a figure which shows an example of operation | movement of 1st embodiment. It is a figure which shows an example of the block diagram of the operation information processing apparatus 80 of 2nd, 3rd embodiment. It is a figure which shows an example of the display part 50 of 2nd embodiment. It is a figure which shows an example of operation | movement of 2nd embodiment. It is a figure which shows an example of operation | movement of 3rd embodiment.

  First, an example of the outline of the present invention will be described with reference to FIG.

  1 is connected to an imaging unit 40 such as a camera and a display unit 50 such as a display. Content such as text and icons is displayed on the display unit 50 used by the user. The imaging unit 40 images the entire user's face and / or eyes, and transmits the captured image to the operation information processing apparatus 80. When a sentence is displayed on the display unit 50, the operation information processing apparatus 80 performs control to detect the movement of the entire face and / or the eyes of the user. Specifically, the operation information processing apparatus 80 extracts only the horizontal (vertical) direction from the movement of the user's face or eyes when the sentence is written vertically (horizontal). The operation information processing apparatus 80 transmits a predetermined operation instruction determined by the extracted movement to the operation execution unit 70.

  According to such a configuration, the operation information processing apparatus 80 according to the present invention can execute an operation that correctly reflects the user's intention. According to the present invention, when the sentence is written horizontally (vertically), the user's eyes frequently move in the horizontal (vertical) direction. Therefore, the movement in the horizontal (vertical) direction is ignored and only the movement in the vertical (horizontal) direction is ignored. By detecting this and associating it with a predetermined operation, it is possible to provide an operation information processing apparatus, method, and program that easily reflects the user's intention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In all the drawings, the same components are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  Each unit constituting the operation information processing apparatus 80 of each embodiment includes a control unit, a memory, a program loaded in the memory, a storage unit such as a hard disk for storing the program, a network connection interface, and the like. Realized by any combination of software. And unless there is particular notice, the realization method and apparatus are not limited.

  The control unit includes a CPU (Central Processing Unit) and the like, operates an operating system to control the entire operation information processing apparatus 80, and reads a program and data into a memory from a recording medium mounted on, for example, a drive device, Various processes are executed according to this.

  The recording medium is, for example, an optical disk, a flexible disk, a magnetic optical disk, an external hard disk, a semiconductor memory, etc., and records a computer program so that it can be read by a computer. The computer program may be downloaded from an external computer (not shown) connected to the communication network.

  In addition, the block diagram used in the description of each embodiment shows a functional unit block, not a hardware unit configuration. These functional blocks are realized by any combination of hardware and software. Further, in these drawings, the components of each embodiment may be described as being realized by one physically coupled device, but the means for realizing the same is not limited thereto. That is, two or more physically separated devices may be connected by wire or wirelessly, and the system of each embodiment may be realized by the plurality of devices.

<Embodiment 1> A configuration example of the first embodiment will be described below.
FIG. 2 shows an example of a block diagram of the operation information processing apparatus 80 of the present embodiment.

  The operation information processing apparatus 80 includes a detection unit 10 and a processing unit 20. Furthermore, an operation condition storage unit 30, a display control unit 60, an operation execution unit 70, an imaging unit 40, and a display unit 50 may be provided.

  The display unit 50 is a display device such as a liquid crystal display, an organic EL (Electro-Luminescence) display, or a CRT (Cathode Ray Tube). On the display unit 50, display objects such as a mouse pointer, a browser, and various contents are displayed.

  When the display object such as the content displayed on the display unit 50 is a sentence, the display control unit 60 has a function of distinguishing whether the sentence is vertical writing or horizontal writing and transmitting the discrimination information to the detection unit 10. The display control unit 60 can determine whether the displayed sentence is vertical writing or horizontal writing, for example, by periodically acquiring a format tag of an HTML (Hyper Text Markup Language) sentence displayed on the browser. . When a plurality of windows are displayed on the display unit 50, the display control unit 60 may acquire only the text format displayed in the top window. The display control unit 60 passes to the detection unit 10 distinction information indicating whether the text displayed on the display unit 50 is vertical writing or horizontal writing. The distinction information may be configured to be generated by the detection unit 10 described later.

  The detection unit 10 includes a detection control unit 100, a face detection unit 101, a face motion detection unit 102, an eye detection unit 103, and an eye motion detection unit 104 (not shown).

  The face detection unit 101 detects the user's face photographed by the imaging unit 40. Face detection performed by the face detection unit 101 can be performed by various methods. For example, the face detection unit 101 includes model information such as contour information relating to the face orientation of a person prepared in advance in the area imaged by the imaging unit 40, part information such as eyebrows, eyes, nose, and mouth, and color information. , And the face and face orientation can be detected by detecting a region having features close to these using template matching. Alternatively, the face detection unit 101 detects a region close to the color of the skin and detects the region as a face, or a method of learning by a teacher signal using a neural network and detecting a region likely to be a face as a face May be used. Alternatively, the face detection unit 101 may attach a marker to the face or forehead and detect the position of the marker with the imaging unit 40.

  The face motion detection unit 102 has a function of determining the movement of the face detected by the face detection unit 101. Face motion detection performed by the face motion detection unit 102 can be performed by various methods. For example, the face motion detection unit 102 continuously acquires data related to the face orientation detected by the face detection unit 101, and detects that the face orientation has changed vertically or horizontally. At this time, the face motion detection unit 102 may determine the orientation of the face by using information such as the position and size of both eyes together. For example, when the positions of both eyes are approximately parallel and approximately the same size, the face motion detection unit 102 determines that the user is facing the front, and when one eye is small, the face is either left or right. It may be determined that it is suitable. Alternatively, the face motion detection unit 102 may determine the movement of the face by detecting that the direction of the nose part or the like has changed. Alternatively, the face motion detection unit 102 may determine the movement of the face by detecting the amount of change of the marker attached to the face or forehead described above. The face motion detection unit 102 captures the movement of the user's face for a certain period of time, obtains time-series data by performing the above matching continuously or at regular intervals, and the face is moving up, down, left, right, or diagonally Can be determined.

  The eye detection unit 103 detects the eyes of the user of the operation information processing apparatus 80 photographed by the imaging unit 40. The eye detection performed by the eye detection unit 103 can be performed by various methods. For example, the eye detection unit 103 extracts the iris or the edge of the pupil from the eyeball image captured by the imaging unit 40 by template matching or the like, performs circle approximation, calculates the center point of the circle, and uses it as the eye position. It may be detected.

  The eye movement detection unit 104 has a function of determining the eye movement detected by the eye detection unit 103. The eye movement detection performed by the eye movement detection unit 104 can be performed by various methods. For example, the eye movement detection unit 104 extracts the iris or pupil edge as described above, performs circle approximation, calculates the center point of the circle, and acquires eye position data detected as the eye position. The line-of-sight direction may be detected from the amount of displacement of the center point. At this time, the eye movement detection unit 104 may determine that the line-of-sight direction has changed only when a certain amount of displacement is detected in order to ignore the fine adjustment of the eyes. The eye movement detection unit 104 captures the eye movement of the user for a certain period of time, and performs the above matching continuously or at regular intervals to obtain time-series data. Can be determined.

  The detection control unit 100 controls the detection of the face motion detection unit 102 and the eye motion detection unit 104 described above based on the distinction information regarding the text format acquired by the display control unit 60. Specifically, when the discrimination information is vertical writing, the detection control unit 100 causes the face motion detection unit 102 and the eye motion detection unit 104 to stop detecting the vertical motion and detect only the horizontal motion. Control. When the discrimination information is horizontal writing, the detection control unit 100 controls the face motion detection unit 102 and the eye motion detection unit 104 to stop detecting the motion in the horizontal direction and detect only the motion in the vertical direction. The discrimination information may be generated by the detection control unit 100 by inputting display data from the display control unit 60, and detection control of the face motion detection unit 102 and the eye motion detection unit 104 described above may be performed based on the discrimination information. good.

  The function of the detection control unit 100 will be described with reference to FIG. FIG. 3 shows that a point in the display unit 50 is moved from the coordinates (x1, y1) to the coordinates (x2, y2) where the eyes of the user looking at the display unit 50 are facing or the front of the face is facing. It is shown that. Here, the method of taking the coordinate system is not particularly limited, but here, a normal Euclidean coordinate system will be used, and the horizontal direction will be described as the x axis and the vertical direction as the y axis.

  When the detection control unit 100 acquires from the display control unit 60 that the text displayed on the display unit 50 is horizontal writing, the face motion detection unit 102 and the eye motion detection unit 104 do not detect horizontal movement. Control. That is, the detection control unit 100 includes a horizontal movement x2-x1 (showing a movement from the x coordinate start point x1 to the end point x2, the same applies hereinafter) and a vertical movement y2-y1 (y coordinate start point y1 to end point. Only the movement y2-y1 in the y direction is detected. Then, the detection control unit 100 passes the vertical movement y2-y1 as a detection result to the processing unit 20 described later.

  In addition, when the detection control unit 100 acquires from the display control unit 60 that the text displayed on the display unit 50 is vertical writing, the face motion detection unit 102 and the eye motion detection unit 104 perform vertical movement. Control not to detect. That is, the detection control unit 100 detects only the vertical movement x2-x1 out of the horizontal movement x2-x1 and the vertical movement y2-y1. Then, the detection control unit 100 passes the lateral movement x2-x1 as a detection result to the processing unit 20 described later.

  Note that the detection control described above is performed only for the movement of the user's face or eyes, and the detection control unit 100 detects the operation as it is without applying a filter with respect to the operation related to the opening and closing of the user's eyes. It is also good.

  In addition, the detection control unit 100 may be a detection target of the detection unit 10 only when the face or eyes move more than a predetermined distance at a predetermined time. For example, a predetermined distance (such as 1 cm; there is no special restriction) within a predetermined time (such as 600 milliseconds, which is not particularly limited) or more is detected so as to be a detection target only when a face or eye is moved. The control unit 100 may control the face motion detection unit 102 and the eye motion detection unit 104. Thereby, the detection part 10 can prevent detecting the fine movement and operation | movement which the face and eyes do not intend by the user.

  The processing unit 20 stores an operation condition storage unit 30 to be described later according to the type of face or eye motion detected by the detection unit 10 (including the size, direction, time, and the like when the motion is a motion). It has a function of referring to and transmitting a predetermined operation instruction corresponding to the operation to the operation execution unit 70. The predetermined operation is, for example, movement or click of a mouse pointer that is a display object displayed on the display unit 50 described later, screen scrolling, mode switching, or other various operations on the computer. The processing unit 20 acquires the type of face or eye movement detected by the detection unit 10 from the detection unit 10 and refers to the operation condition storage unit 30. When the processing unit 20 determines that the condition defined in the operation condition storage unit 30 is satisfied, the processing unit 20 transmits a predetermined processing operation instruction corresponding to the condition to the operation execution unit 70. The operation condition may be included in the processing unit 20 instead of the operation condition storage unit 30. The processing unit 20 may generate an operation instruction by calculating from the user's face or eye movement obtained from the detection unit 10 and output the operation instruction to the operation execution unit 70.

  The operation condition storage unit 30 stores a face or eye movement and a predetermined operation in association with each other. Here, the action of the face or eyes refers to the movement of the entire face of the user, only the eye part or both, or other actions. Here, the movement is a movement of a certain distance or more of the entire face and / or the eyes where the starting point and the ending point are determined. For example, when the front of the face or the viewpoint of the eyes stays at a certain position for a certain time, the position is the start point or the end point. Specifically, the face detection unit 101 and the eye detection unit 103 detect the front position of the user's face or the position of the eye viewpoint and the time that the user stays at the position, thereby determining the start and end points of the user's movement. It may be detected. In addition, the other motions include a left-right movement of the face determined by blinking eyes or template matching. For example, the operation condition storage unit 30 stores, as an operation, a case where the coordinate position of the point in the display unit 50 where the eyes are facing or the front of the face is facing is changed within a predetermined time. In addition, a predetermined operation for the operation is stored in association with the operation.

  FIG. 4 is a diagram illustrating an example of the operation condition storage unit 30. Here, the operation condition storage unit 30 has an “eye movement” column and an “operation” column. As will be described later, the operation condition storage unit 30 may include a “face motion” column instead of the “eye motion” column, or a “face motion” column in addition to the “eye motion” column. Also good. Hereinafter, the operation condition storage unit 30 in FIG. 4 will be described as an example.

  The “eye movement” column stores the movement of the eye position in the horizontal or vertical direction within a predetermined time or the open / closed state of the eye as the eye movement. For example, “y2−y1: 5 cm or more: within 600 milliseconds” indicates a state in which the user's eyes have moved in the vertical direction by 1 cm or more within 600 milliseconds. “X2−x1: 5 cm or more: within 600 milliseconds” indicates a state where the user's eyes have moved laterally by 5 cm or more within 600 milliseconds. Further, “close and open left eye for one second” indicates that the user closes and opens the left eye for one second.

  The “operation” column stores the contents of the operation instruction that the processing unit 20 outputs to the operation execution unit 70 in response to the operation of the “eye movement” column performed by the user. For example, referring to FIG. 4, when the “eye movement” column is “y2−y1: 5 cm or more: within 600 milliseconds”, the corresponding “operation” column is “c * (y2−y1): screen”. Scroll. " This is an operation instruction for scrolling the screen vertically by c * (y2-y1) when the user's eye movement moves 5 cm or more y2-y1 within 600 milliseconds in the vertical direction. It shows that the unit 20 outputs to the operation execution unit 70. Here, c is a constant. In addition, the user can arbitrarily set c as a parameter.

  In FIG. 4, the “operation” column corresponding to “the left eye is closed and opened for one second” in the “eye movement” column is “mouse left click”. This indicates that the processing unit 20 outputs an operation instruction meaning a left click of the mouse to the operation execution unit 70 when the user's eye movement is to close and open the left eye for 1 second. The “operation” column corresponding to “close both eyes for 5 seconds” in the “eye movement” column is “discontinuation of detection function”. This means that when the user closes both eyes for 5 seconds, the processing unit 20 outputs to the operation execution unit 70 an operation instruction that means to cancel the detection function of the detection unit 10. Similarly, when the “eye movement” column is “repeats blinking of both eyes three times”, the processing unit 20 outputs an operation instruction meaning switching between the mouse and pen tablet mode corresponding thereto to the operation execution unit 70. To do. When the “eye movement” column is “close the right eye for 2 seconds in the pen tablet state”, the processing unit 20 outputs to the operation execution unit 70 an operation instruction that means changing the pen pressure of the corresponding pen tablet. . If the “eye movement” column is “move left eye while closing right eye in pen tablet state”, processing unit 20 outputs a corresponding drawing operation instruction to the pointing destination to operation execution unit 70.

  Note that, regarding the contents of the “eye movement” column, the contents of the above-described eye movement, the movement distance of the eye, the predetermined time during which the movement is performed, the operation corresponding thereto, and the like may be appropriately set by the user. For example, the operation condition storage unit 30 sets “c * (x2−x1): move the mouse pointer” to the “operation” column corresponding to the “eye movement” column “y2−y1: 5 cm or more: within 600 milliseconds”. The mouse pointer movement may be stored as a predetermined operation.

  In addition, the user may provide a plurality of “eye movement” columns in the operation condition storage unit 30 through an input unit (not shown) or the contents of the “operation” columns corresponding to the plurality of movements. It may be changed as appropriate. For example, the user provides two “eye movement” columns in the operation condition storage unit 30, one of which is set to “close the left eye” and the other one corresponding to “moves the line of sight of the right eye: 1 cm It is possible to set “drag” in one of the “operation” columns corresponding thereto. Thus, when the user closes the left eye and moves the line of sight of the right eye by 1 cm or more, the processing unit 20 may output an operation instruction that means mouse dragging to the operation execution unit 70.

  It should be noted that the user may provide a “face motion” column for storing the face motion via an input unit (not shown) or in advance, instead of the “eye motion” column of the operation condition storage unit 30. In this case, the user's face motion is stored in the “face motion” column, and a predetermined operation corresponding to the face motion is stored in the “operation” column, as in the case of the “eye motion” column described above. Is done. In this case, the processing unit 20 outputs a predetermined operation instruction corresponding to the facial motion to the operation execution unit 70.

  FIG. 5 is a diagram illustrating another example of the operation condition storage unit 30. As illustrated in FIG. 5, the operation condition storage unit 30 may include a “face motion” column that stores a face motion together with an “eye motion” column. In this case, the operation condition storage unit 30 can associate the detection result of both the eye movement and the face movement with a predetermined operation. For example, the operation condition storage unit 30 sets one of the “eye motion” columns to “close the left eye”, and one of the corresponding “face motion” columns to “move the face left and right”, The condition can be “and” and one of the “operation” columns corresponding to the condition can be “browser return”. As a result, if the user swings his / her face left and right while closing his left eye, the operation execution unit 70 can execute the function of returning the browser. Further, as shown in the second row of FIG. 5, the operation condition storage unit 30 sets both the “eye motion” column and the “face motion” column to “x2−x1: 1 cm or more: within 600 milliseconds” The column can also be “or”. In this case, the processing unit 20 determines that the condition is satisfied when the movement of the face of 1 cm or more is detected within 600 milliseconds in either the eye movement or the face movement, and a predetermined operation corresponding to the condition is satisfied. The instruction may be output to the operation execution unit 70.

  The imaging unit 40 is a CCD (Charge Coupled Device) camera, an analog or digital video camera, and the like, and is configured to have a function of photographing the entire face of the user, only the eye part, or both. The operation information processing apparatus 80 may include a face detection imaging device and an eye detection imaging device separately. In addition, the imaging unit 40 is preferably installed at a position where it is easy to photograph the user, such as an upper part of the display unit 50 described later. However, if the user is in a position where the text on the display unit 50 can be viewed, The installation location is not limited. The imaging unit 40 may further include a user position detection function, and the imaging unit 40 may be configured to detect the user's position and automatically face the user direction.

  The operation execution unit 70 receives an operation instruction from the processing unit 20 and executes a predetermined operation corresponding to the operation instruction. For example, the operation execution unit 70 processes an operation instruction instead of an OS (Operating System) generally inputting a user operation instruction (selection of an icon, a button, dragging a scroll bar, a command, etc.) from a mouse or the like. The predetermined operation corresponding to the operation instruction acquired from the unit 20 is executed. That is, the operation execution unit 70 is an operation command execution unit or a user interface unit that is generally provided in an OS extended to accept an operation instruction from the processing unit 20.

  Next, the operation of the invention in this embodiment will be described with reference to FIG. First, the imaging unit 40 that captures a user at a position where the text displayed on the display unit 50 can be viewed captures the entire user's face and / or eyes, and the captured image is sent to the detection unit 10. Transmit (S1 in FIG. 6). When the text is displayed on the display unit 50, the display control unit 60 transmits to the detection unit 10 distinction information indicating whether the text displayed on the display unit 50 is vertical writing or horizontal writing (S2 in FIG. 6). . Next, the detection unit 10 changes the detection method based on the discrimination information (S3 in FIG. 6). Specifically, when the discrimination information is vertical writing, the face motion detection unit 102 or the eye motion detection unit 104 extracts only horizontal movement by the above-described method, and when the discrimination information is horizontal writing. Extract only vertical movement. Next, the detection unit 10 transmits the detected result to the processing unit 20 (S4 in FIG. 6). The processing unit 20 refers to the conditions stored in the operation condition storage unit 30, and transmits a predetermined operation instruction corresponding to the received detection result to the operation execution unit 70 (S5 in FIG. 6).

  Next, the operation will be described based on a specific example with reference to FIG. Here, a case where the operation condition storage unit 30 is as shown in FIG. 4 and the user is browsing the text displayed on the browser will be described as an example. First, the imaging unit 40 images the user who is reading the text displayed on the display unit 50, and transmits the captured image to the detection unit 10 (S1 in FIG. 6). Here, when the sentence is horizontally written, the display control unit 60 transmits the discrimination information indicating that the displayed sentence is horizontally written to the detecting unit 10 (S2 in FIG. 6). When the detection unit 10 acquires that the distinction information is horizontal writing, the detection unit 10 detects only the vertical movement of the user (S3 in FIG. 6). Therefore, when the user moves his / her eyes in the horizontal direction, the detection unit 10 detects only movement in the vertical direction, and thus the movement of the user's eyes is not detected and ignored. On the other hand, when the user moves his / her eyes in the vertical and vertical directions, for example, 10 cm within 600 milliseconds, the detection unit 10 detects only the vertical movement, and thus the vertical movement of the user's eyes is detected by the detection unit 10. Detected. And the detection part 10 transmits this detection result to the process part 20 (S4 of FIG. 6). The processing unit 20 refers to the operation condition storage unit 30. Here, since the operation condition storage unit 30 is described in FIG. 4, when the user moves his / her eyes 10 cm vertically within 600 milliseconds, “y2−y1: The condition “5 cm or more: within 600 milliseconds” is satisfied. Therefore, the processing unit 20 transmits “C * (y2−y1): screen scroll” (for example, scrolling the screen by 10 cm when C is 1) corresponding to this as an operation instruction to the operation execution unit 70 (FIG. 6). S5).

  In the present embodiment, an operation information processing apparatus 80, a method, and a program that execute an operation that correctly reflects the user's intention are provided. In particular, when the text is written horizontally (vertically), the user's face and eyes move frequently in the horizontal (vertical) direction, so the movement in the horizontal (vertical) direction is ignored and only the movement in the vertical (horizontal) direction is ignored. By detecting and associating with a predetermined operation, an operation that correctly reflects the user's intention can be executed.

  <Embodiment 2> The operation information processing apparatus 80 of the present embodiment is based on Embodiment 1 and differs in the following points. First, an example of the outline of the present embodiment will be described.

  The operation information processing apparatus 80 according to the present embodiment determines whether or not the display object that the user is interested in is a sentence. If the display object is a sentence, the operation information processing apparatus 80 determines whether the sentence is a vertical writing or a horizontal writing. Controls eye movement detection.

  According to this, it is possible to perform detection control of the user's face or eyes according to the format of the text that the user pays attention to.

  Below, the structure of the operation information processing apparatus 80 of this embodiment is demonstrated.

  FIG. 7 shows an example of a block diagram of the present embodiment. The operation information processing apparatus 80 includes a detection unit 10, a determination unit 200, and a processing unit 20. The operation information processing apparatus 80 may further include an operation condition storage unit 30, a display control unit 60, an operation execution unit 70, an imaging unit 40, and a display unit 50. Since the processing unit 20, the operation condition storage unit 30, the display control unit 60, the operation execution unit 70, the imaging unit 40, and the display unit 50 are substantially the same as those in the first embodiment, detailed description thereof is omitted.

  The determination unit 200 includes an eye detection unit 103, an eye movement detection unit 104, and a detection control unit 100 (not shown). Since the configurations and operations of the eye detection unit 103 and the eye movement detection unit 104 are the same as those in the first embodiment, detailed description thereof is omitted. The determination unit 200 in the present embodiment is configured to detect a point of interest indicating which position on the display unit 50 the user is paying attention to. Here, since the calculation of the attention point can be performed by various known methods, the description is omitted. For example, the detection control unit 100 activates the eye detection unit 103 at regular time intervals, and the eye detection unit 103 acquires position information indicating where the user's attention point is. Alternatively, the detection control unit 100 activates the eye detection unit 103 when the eye movement detection unit 104 moves the user's eyes over a predetermined distance, and the eye detection unit 103 determines where the user's attention point is located. Get some location information.

  The determination unit 200 is configured to acquire position information of a user's attention point and also acquire information on one or more display objects displayed on the display unit 50 from the display control unit 60. Here, the information of the display object is, for example, type information indicating whether or not the type of display object is a sentence, and when the type of display object is a sentence, distinction information indicating whether the sentence is vertical writing or horizontal writing, This is the position information where the display object on the display unit 50 exists. The information on the display object may be directly generated by the determination unit 200 by inputting display data from the display control unit 60.

  The determination unit 200 compares the position information of the user's attention point and the information of the display object at the attention point position, and determines the type of the display object at the user's attention point position. Although the determination method is not particularly limited, an example will be described with reference to FIG. FIG. 8 shows an example of the screen of the display unit 50. In the rectangle defined by the coordinates (a1, b1) and (a2, b2), the text as the display object 1 is displayed, and the coordinates (c1, d1) And (c2, d2) indicate that a display object other than the text which is the display object 2 is displayed in the rectangle defined by (c2, d2). The coordinates (x1, y1) are the user's attention points at a certain time, and (x2, y2) are the user's attention points at another time.

  A case where the user's attention point is at coordinates (x1, y1) will be described. The determination unit 200 acquires that the user's attention point is at the coordinates (x1, y1), and a sentence is displayed in the rectangle defined by the coordinates (a1, b1) and (a2, b2). This is acquired from the display control unit 60.

  The determination unit 200 calculates that a1 <x1 <a2 and b1 <y1 <b2, and coordinates (x1, y1) that are the user's attention points are defined by coordinates (a1, b1) and (a2, b2). Therefore, it is determined that the user's attention point is on the sentence that is the display object 1. In this case, since the user is paying attention to the sentence, the determination unit 200 further acquires the discrimination information indicating whether the sentence of the display object 1 is vertical writing or horizontal writing from the display control unit 60, and The discrimination information is transmitted to the detection unit 10.

  When the user's attention point is at the coordinates (x2, y2), the determination unit 200 determines that the user's attention point is on the display object 2 other than the sentence by the same steps. In this case, the determination result is not transmitted to the detection unit 10.

  The detection unit 10 includes a detection control unit 100, a face detection unit 101, a face motion detection unit 102, an eye detection unit 103, and an eye motion detection unit 104 (not shown). Since the configuration and operation of each detection unit are substantially the same as those in the first embodiment, the details are omitted. The detection unit 10 in the present embodiment is configured to acquire the discrimination information from the determination unit 200 and change the detection method according to the discrimination information. Specifically, when the user's attention point is on the sentence and the discrimination information that the sentence is written vertically (horizontal) is acquired, the horizontal (vertical) direction of the user's face or eye movement is acquired. Only the movement is detected by the detection unit 10. On the other hand, the detection part 10 detects a user's operation | movement as it is, without applying a filter, when discriminating information is not received from the determination part 200. FIG. Therefore, the detection unit 10 can control the detection only when the user is looking at the sentence. Further, when a plurality of sentences are displayed on the display unit 50, the detection unit 10 can perform control according to the sentence format. For example, when a vertically written sentence and a horizontally written sentence are displayed on the display unit 50 at the same time, the detecting unit 10 only focuses on the entire face or eye part of the user when the user is paying attention to the vertically written sentence. Only the horizontal movement is detected. When the user is paying attention to the horizontally written sentence, the detection unit 10 detects only the vertical movement among the movements of the entire face or only the eyes of the user.

  Thereafter, as in the first embodiment, the processing unit 20 refers to the operation condition storage unit 30 and transmits an operation instruction to the operation execution unit 70, and thus detailed description thereof is omitted.

  FIG. 9 shows an operation flow of the operation information processing apparatus 80 in the present embodiment. The determination unit 200 detects the point-of-interest coordinates on the display unit 50 from the image of the user's eyes obtained from the imaging unit 40 (S1 in FIG. 9). Specifically, the detection control unit 100 activates the eye detection unit 103 at regular time intervals, and the eye detection unit 103 acquires position information indicating where the user's attention point is. Alternatively, the detection control unit 100 activates the eye detection unit 103 when the eye movement detection unit 104 moves the user's eyes for a predetermined distance or more, and the eye detection unit 103 detects the user's attention point. You may acquire the positional information on where. The determination unit 200 determines whether the display object at the attention point is a sentence or a display object other than a sentence based on the detection result and the information of the display object from the display control unit 60. The determination unit 200 does not transmit the discrimination information to the detection unit 10 when the display object at the point of interest is other than a sentence (S2 in FIG. 9). On the other hand, when the display object at the point of interest is a sentence, the determination unit 200 acquires the distinction information about whether the sentence is vertical writing or horizontal writing from the display control unit 60 and sends the distinction information to the detection unit 10. Transmit (S3 in FIG. 9). The detection unit 10 determines whether or not the identification information is acquired from the determination unit 200. When the detection unit 10 obtains the discrimination information from the determination unit 200 (Y in S4 in FIG. 9) and receives the discrimination information that the sentence is written vertically (horizontal), the entire face of the user or Only the movement in the horizontal (vertical) direction is detected among the movements of the eyes, and the detection result is transmitted to the processing unit (S5 in FIG. 9). On the other hand, when the detection unit 10 does not acquire the discrimination information from the determination unit 200 (N in S4 in FIG. 9), the detection unit 10 detects the user's operation without filtering and transmits the detection result to the processing unit. (S6 in FIG. 9). The processing unit 20 acquires the detection content from the detection unit 10, and refers to the content of the operation condition storage unit 30 to give an operation execution instruction to the operation execution unit 70 (S7 in FIG. 9).

  In the present embodiment, an operation information processing apparatus, method, and program for executing an operation that correctly reflects the user's intention are provided. In particular, since the detection unit 10 determines the detection of the user's face and eyes according to the display object at the position of the user's attention on the display unit 50, it is possible to execute an operation that correctly reflects the user's intention. it can.

  <Third Embodiment> An operation information processing apparatus 80 according to the present embodiment is based on the second embodiment and is different in the following points. First, an example of the outline of the present embodiment will be described.

  The operation information processing apparatus 80 according to the present embodiment determines whether or not the display object that the user is interested in is a sentence. If the display object is a sentence, the user's face or eyes are determined according to whether the sentence is written vertically or horizontally. The detection unit 10 controls the detection of the movements. At this time, when the display object that the user is interested in is a sentence, the detection unit 10 stops detecting the movement of the eyes of the user and only detects the movement of the entire face. At that time, if the sentence is written vertically, only the horizontal movement is extracted from the movement of the entire face. If the sentence is written horizontally, only the vertical movement is extracted from the movement of the entire face.

  Since the configuration of the operation information processing apparatus 80 of the present embodiment is substantially the same as that of the second embodiment except for the function of the detection unit 10, only the function of the detection unit 10 will be described.

  The detection unit 10 includes a detection control unit 100, a face detection unit 101, a face motion detection unit 102, an eye detection unit 103, and an eye motion detection unit 104. Since the functions and configurations of each of these detection units are substantially the same as those in the first embodiment, detailed description thereof is omitted. In the present embodiment, the detection control unit 100 controls to stop the detection function of the eye movement detection unit 104 when the determination result that the user's attention point is a sentence is acquired from the determination unit 200. Thereby, when the user is paying attention to the sentence, the detection unit 10 does not detect the movement of the eye part, but detects the movement of the entire face. Here, the detection unit 10 does not necessarily have to stop the detection of the eye detection unit 103. That is, of the eye movements, only the eye movement may not be detected, and other movements (open / closed state of the eye such as whether the eyes are blinking) may be continuously detected. In this case, for example, the detection unit 10 can detect that both eyes are closed for 5 seconds and send the result to the processing unit 20. Therefore, taking the operation condition storage unit 30 in FIG. 4 as an example, the processing unit 20 transmits “discontinuation of detection function” corresponding to “close both eyes for 5 seconds” to the operation execution unit 70 so that the operation execution unit 70 It is also possible to perform an operation to stop the function of the detection unit 10 itself.

  FIG. 10 shows an operation flow of the operation information processing apparatus 80 in the present embodiment. In the operation flow of FIG. 10, the steps up to S4 are the same as those in the second embodiment, and thus detailed description thereof is omitted. S4 and subsequent steps will be described. The detection unit 10 determines whether or not the identification information is acquired from the determination unit 200. When the detection unit 10 acquires the discrimination information from the determination unit 200 (Y in S4 in FIG. 10), and when the discrimination unit 200 acquires the discrimination information that the sentence is vertical (horizontal) writing, Then, the detection of the movement of the user's eyes is stopped, and only the movement in the horizontal (vertical) direction is detected from the movement of the entire face and transmitted to the processing unit 20 (S5 in FIG. 10). On the other hand, when the detection unit 10 does not acquire the discrimination information from the determination unit 200 (N in S4 in FIG. 10), the detection unit 10 detects the user's operation as it is without applying a filter and sends the detection result to the processing unit. Transmit (S6 in FIG. 10). The processing unit 20 acquires the detection content from the detection unit 10, and refers to the content of the operation condition storage unit 30 to give an operation execution instruction to the operation execution unit 70 (S7 in FIG. 10).

  Next, the operation will be described based on a specific example with reference to FIG. Here, the case where the operation condition storage unit 30 is as shown in FIG. 5 will be described by taking as an example the case where the user is browsing the horizontally written text displayed on the browser. First, the imaging unit 40 images a user who is viewing the content displayed on the display unit 50 and distributes the captured image to the determination unit 200. The determination unit 200 acquires the user's attention point information on the display unit 50, acquires information on the display object from the display control unit 60, and determines whether the display object at the user's attention point on the display unit 50 is a sentence. Is transmitted to the detection unit 10. In this example, since the determination unit 200 determines that the content focused on by the user is a sentence and is written horizontally, the determination unit 200 transmits the distinction information to the detection unit 10 (S3 in FIG. 10). When the detection unit 10 acquires that the sentence distinction information is horizontal writing, the detection unit 10 stops detecting the movement of the eyes of the user, detects only the vertical movement of the entire face, and sends the detection result to the processing unit 20. Transmit (S4, S5 in FIG. 10). Therefore, when the user moves his / her eyes only up / down / left / right, the detection unit 10 does not detect the movement of the user's eyes. On the other hand, when the user moves the entire face in the vertical and vertical directions, for example, 10 cm within 600 milliseconds, the vertical movement of the user's face is detected by the detection unit 10. And the detection part 10 transmits this detection result to the process part 20 (S5 of FIG. 10). The processing unit 20 refers to the operation condition storage unit 30. Here, since the operation condition storage unit 30 is described in FIG. 5, when the user moves the face 10 cm vertically within 600 milliseconds, “y2-y1: The condition “5 cm or more: within 600 milliseconds” is satisfied. Therefore, the processing unit 20 transmits “C * (y2−y1): screen scroll” (for example, scrolling the screen by 10 cm if C is 1) to the operation execution unit 70 as an operation instruction (FIG. 10). S7).

  In the present embodiment, an operation information processing apparatus, method, and program for executing an operation that correctly reflects the user's intention are provided. In particular, when the display object on which the user pays attention on the display unit 50 is a sentence, the detection unit 10 stops detecting the movement of the eyes of the user and only detects the movement of the entire face, and the sentence is vertically (horizontal). In the case of writing, only the movement in the horizontal (vertical) direction is extracted from the movement of the entire face. In this way, the user's intention is to detect only the movement in the horizontal (vertical) direction of the entire face and associate it with a predetermined operation without detecting the movement of the eye part that frequently moves when the user reads a sentence. Operations that reflect correctly can be executed.

  While the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

DESCRIPTION OF SYMBOLS 10 Detection part 20 Processing part 30 Operation condition memory | storage part 40 Imaging part 50 Display part 60 Display control part 70 Operation execution part 80 Operation information processing apparatus 100 Detection control part 101 Face detection part 102 Face motion detection part 103 Eye detection part 104 Eye movement Detection unit 200 determination unit

Claims (10)

Acquisition of distinction information on whether a sentence as a display object displayed on the display means is vertical writing or horizontal writing, and imaging by the imaging means for imaging the face or eyes of the user existing at a position where the user can see the text Image input and detection of the user's face or eye movement from the image. When the discrimination information is vertical writing, a horizontal movement is extracted from the movement, and the discrimination information is In the case of horizontal writing, detection means for extracting and outputting the vertical movement of the above operation,
An operation information processing apparatus comprising: a processing unit that transmits a predetermined operation instruction determined by the movement output by the detection unit to an operation unit that executes the operation instruction. An attention point within the display surface of the display means of the user's line of sight is detected from the image, and when the display object displayed at the attention point is a sentence, the distinction information is acquired and the distinction information is A determination unit that transmits to the detection unit and does not transmit the distinction information when the display object displayed at the point of interest is not a sentence;
When the discrimination information is received from the determination unit, the detection unit extracts a movement in the horizontal direction from the operation when the discrimination information is vertical writing, and when the discrimination information is horizontal writing, The operation information processing apparatus according to claim 1, wherein a motion in the vertical direction is extracted and output, and the operation is output when the distinction information is not received.   The detection unit extracts only the movement of the user's face when the discrimination information is received, and extracts only the user's eyes or the movement of the face and eyes when the discrimination information is not received. Item 3. The information processing apparatus according to Item 2. Operation condition storage means for storing the operation output by the detection means and a predetermined operation instruction in association with each other;
The display means;
The imaging means provided near the display surface of the display means;
Display control means for displaying the display object on the display means and outputting the distinction information when the display object is a sentence;
The operating means;
The processing means for transmitting a predetermined operation instruction stored in the operation condition storage means in association with the movement output by the detection means to the operation means. Operation information processing device. Acquisition of distinction information on whether a sentence as a display object displayed on the display means is vertical writing or horizontal writing, and imaging by the imaging means for imaging the face or eyes of the user existing at a position where the user can see the text Image input and detection of the user's face or eye movement from the image. When the discrimination information is vertical writing, a horizontal movement is extracted from the movement, and the discrimination information is In the case of horizontal writing, a detection step of extracting and outputting a vertical movement out of the operations,
An operation information processing program that causes a computer to execute a processing step of transmitting a predetermined operation instruction determined by the movement output in the detection step to an operation means that executes the operation instruction. An attention point within the display surface of the display means of the user's line of sight is detected from the image, and the distinction information is acquired when the display object displayed at the attention point is a sentence, and is displayed at the attention point A determination step of not acquiring the distinction information when the displayed object is not a sentence;
In the determination step, when the discrimination information is acquired, if the discrimination information is vertical writing, a horizontal motion is extracted from the operation, and if the discrimination information is horizontal writing, a vertical motion of the operation is extracted. In the case of extracting and outputting a motion and not obtaining the distinction information, the detection step of outputting the motion;
6. The operation information processing program according to claim 5, which causes the computer to execute.   In the determining step, when the discrimination information is acquired, only the movement of the user's face is extracted, and when the discrimination information is not acquired, only the user's eyes or the movement of the face and eyes is extracted. The operation information processing program according to claim 6, which causes the computer to execute a detection step. Acquisition of distinction information on whether a sentence as a display object displayed on the display means is vertical writing or horizontal writing, and imaging by the imaging means for imaging the face or eyes of the user existing at a position where the user can see the text Image input and detection of the user's face or eye movement from the image. When the discrimination information is vertical writing, a horizontal movement is extracted from the movement, and the discrimination information is In the case of horizontal writing, a detection step of extracting and outputting a vertical movement out of the operations,
An operation information processing method comprising: a processing step of transmitting a predetermined operation instruction determined by the movement output in the detection step to an operation unit that executes the operation instruction. An attention point within the display surface of the display means of the user's line of sight is detected from the image, and the distinction information is acquired when the display object displayed at the attention point is a sentence, and is displayed at the attention point A determination step of not acquiring the distinction information when the displayed object is not a sentence;
In the determination step, when the discrimination information is acquired, if the discrimination information is vertical writing, a horizontal motion is extracted from the operation, and if the discrimination information is horizontal writing, a vertical motion of the operation is extracted. In the case of extracting and outputting a motion and not obtaining the distinction information, the detection step of outputting the motion;
The operation information processing method according to claim 8.   In the determining step, when the discrimination information is acquired, only the movement of the user's face is extracted, and when the discrimination information is not acquired, only the user's eyes or the movement of the face and eyes is extracted. The operation information processing method according to claim 9, further comprising a detection step.

Images