JP7215254B2 - Information processing device, display control method, and program - Google Patents

Description

The present invention relates to an information processing device, a display control method, and a program.

2. Description of the Related Art Conventionally, there is already known a technique of temporarily displaying an edit mode image around a writing position when writing or selecting a menu on a screen of an electronic blackboard with a predetermined pen.

With this technology, when a pen is brought into contact with the screen and writing is performed, an edit mode image is temporarily displayed around the writing position, and the writer can confirm the type of editing mode at the writing position. (See Patent Document 1).

However, with the conventional technique of temporarily displaying an edit mode image around the writing position, the menu is displayed around the writing position while the writer is writing. However, the menu will not be displayed unless you have started writing. Therefore, since the scribe does not know the type of edit mode until he or she starts writing, there is room for improvement in terms of operability.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an information processing apparatus, a display control method, and a program capable of improving the operability of a menu such as a writing menu by an operator. do.

In order to solve the above-described problems and achieve the object, an information processing apparatus according to the present invention includes detection means for detecting a line-of-sight position of an operator on a screen, the line-of-sight position detected by the detection means, When the position of the menu on the screen is distant, control is performed to bring the menu closer to the line-of-sight position, and when the line-of-sight position detected by the detection means is out of the screen, the position of the menu is moved. and control means for performing control to return to the original position .

According to the present invention, it is possible to improve the operability of menus by the operator.

FIG. 1 is a diagram illustrating a schematic configuration of an electronic blackboard according to an embodiment. FIG. 2 is a diagram showing an example of the functional block configuration for menu display control of the electronic blackboard. FIG. 3 is a flowchart showing an example of menu display control. FIG. 4 is a conceptual diagram showing movement of the menu area showing an example of the display state of the menu within the screen. FIG. 5 is a conceptual diagram showing an enlarged display of icons and the like in a menu area showing an example of the display state of the menu within the screen.

Hereinafter, embodiments of an information processing apparatus, a display control method, and a program according to the present invention will be described in detail with reference to the accompanying drawings. Although an example in which the information processing apparatus according to the present invention is applied to an electronic blackboard information processing apparatus is shown as an embodiment of the information processing apparatus according to the present invention, the information processing apparatus is not limited to the electronic blackboard information processing apparatus. The information processing apparatus according to the present invention can be suitably applied to an information processing apparatus that outputs screen information on a screen with a wide display area, a PC (Personal Computer), a tablet terminal, and the like.

(Embodiment)
FIG. 1 is a diagram illustrating a schematic configuration of an electronic blackboard according to an embodiment. As shown in FIG. 1, the electronic whiteboard 2 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an SSD (Solid State Drive) 204, a network I/ F205 and an external device connection I/F (Interface) 206 are provided.

The CPU 201 executes applications and controls the entire operation of the electronic blackboard 2 . The ROM 202 mainly stores programs executed by the CPU 201 at startup. A RAM 203 serves as a work area when the CPU 201 executes an application. The SSD 204 is a non-volatile memory that stores applications, data, etc. for the electronic blackboard. Applications for electronic blackboards include programs for line-of-sight detection and menu display control. A network I/F 205 performs processing based on a communication protocol when communicating with a server or the like via a network. Note that the network is a LAN (Local Area Network), the Internet, or the like.

The external device connection I/F 206 is an interface for connecting various external devices. The external device is, for example, an external device (microphone 240, speaker 250, etc.). Also, for example, a detachable external memory is connected, and writing to or reading from the external memory is performed. The external memory is, for example, a USB (Universal Serial Bus) memory 230 or an SD card.

The electronic blackboard 2 also includes a capture device 211, a GPU (Graphics Processing Unit) 212, a display controller 213, a contact sensor 214, a sensor controller 215, an electronic pen controller 216, a line-of-sight detection camera controller 217, a photographing camera controller 218, a short distance A communication circuit 219 , an antenna 219 a of the short-range communication circuit 219 , a power switch 222 and selection switches 223 are provided.

The capture device 211 takes in (captures) the image displayed on the display device by the external PC 270 . The GPU 212 is a processor dedicated to drawing that calculates the pixel value of each pixel on the display 280 . The display controller 213 outputs the image created by the GPU 212 to the display 280 .

The sensor controller 215 detects the position (position on the screen) touched by the pen or finger based on the signal output from the contact sensor 214 . The contact position is detected by an infrared shielding method. In addition to the infrared blocking method, a capacitance method that detects the contact position by changing the capacitance, a resistive film method that detects the contact position by changing the voltage of two opposing resistive films, an electromagnetic induction method, etc. , the detection method may be selected as appropriate.

Electronic pen controller 216 communicates with electronic pen 290 . For example, the electronic pen controller 216 acquires information indicating the orientation and position from the electronic pen 290 to determine the state of use of the electronic pen, the position at which the electronic pen 290 is being used, and the like.

Electronic pen 290 includes a gyro sensor or the like, and communicates with electronic pen controller 216 . When the electronic pen 290 detects a change in posture, such as by being held by a writer, it transmits a signal to the electronic pen controller 216 . For example, the electronic pen controller 216 and the electronic pen 290 each include communication means (transmitting means and receiving means) for performing communication by short-range wireless communication or the like, and the electronic pen 290 transmits a signal indicating a change in posture from the transmitting means, In the electronic pen controller 216, the receiving means receives the signal and detects that the posture has changed.

The line-of-sight detection camera controller 217 controls the camera (the line-of-sight detection camera 220) and acquires the line-of-sight information of the person from the captured image. For line-of-sight detection, there are, for example, a line-of-sight detection method using the inner corner of the eye as a reference point and the iris as a moving point, and a line-of-sight detection method using the corneal reflection as a reference point and the pupil as a moving point. When detecting the line of sight by the former method, a visible camera is used as the camera 220 for line of sight detection. When the line of sight is detected by the latter method, an infrared irradiation device and an infrared camera are used in combination as the line of sight detection camera 220 . The line-of-sight detection camera controller 217 detects in real time the direction of the person's line of sight with respect to the line-of-sight detection camera 220 from the frame images output from the line-of-sight detection camera 220 by the above method or other known line-of-sight detection methods. calculate.

The photographing camera controller 218 controls the camera (the photographing camera 210) and acquires a photographed image. It is desirable that the photographing camera 210 and the line-of-sight detection camera 220 are synchronously photographed. Further, by arranging the photographing camera 210 and the line-of-sight detection camera 220 side by side and photographing in the same direction and the same field angle, it is possible to omit the transformation processing of each coordinate space.

The imaging camera 210 has an imaging sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) provided with a visible light filter such as RGB, and transfers a frame image obtained by imaging an imaging range at a predetermined frame rate. do.

The line-of-sight detection camera 220 is, for example, an infrared camera, and has an infrared irradiation device. Transfers frame images captured in the shooting range at the frame rate.

The short-range communication circuit 219 is a communication circuit such as NFC (Near Field Communication) or Bluetooth (registered trademark). The power switch 222 is a switch for switching ON/OFF of the power of the electronic blackboard 2 . The selection switches 223 are, for example, a group of switches for adjusting the brightness and color of the display on the display 280 . A bus line 210 electrically connecting the CPU 201 and the like is an address bus, a data bus, or the like.

(function block)
The CPU 201 implements various functions of the electronic blackboard 2 by executing applications stored in the SSD 204 or the like. Various functions of the electronic blackboard 2 include basic functions such as input acceptance and display, as well as various editing functions. These editing functions are grouped and displayed as icons in a partial menu area of the display screen of the display 280 . The operator can select a function from the icons in the menu area and write with the selected function. In the present embodiment, mainly the function of controlling the display of the displayed menu area will be described. Note that the above operator will be described as a writer in order to show an example of writing.

FIG. 2 is a diagram showing an example of the configuration of functional blocks for menu display control of the electronic blackboard 2. As shown in FIG. FIG. 2 shows a line-of-sight calculation unit 11, a determination unit 12, and a menu display control unit 13 as functions of the control unit 10 that controls menu display.

The line-of-sight calculation unit 11 calculates which coordinate of the two-dimensional coordinate plane set on the screen of the display 280 corresponds to the line of sight. By setting conditions in advance, the line-of-sight calculation unit 11 can perform a minimum line-of-sight calculation by using real-time line-of-sight information from the camera controller 217 for line-of-sight detection. This is an example of more accurate line-of-sight calculation using real-time captured images.

For example, the line-of-sight calculation unit 11 calculates the line-of-sight of the writer on the screen of the display 280 based on real-time line-of-sight information from the line-of-sight detection camera controller 217 and real-time captured images (frame images) from the photographing camera controller 218. The position (coordinates) of is obtained at any time. Specifically, it is assumed that the line-of-sight calculation unit 11 is preset with information about the installation positions and orientations of the photographing camera 210 and the line-of-sight detection camera 220 . The line-of-sight calculation unit 11 analyzes the distance of the writer in the direction perpendicular to the screen of the display 280 and the coordinates of the face (especially the eyes) on the two-dimensional coordinate plane of the screen from the image captured by the camera 210. calculate. The scribe may be identified from image analysis or by receiving a signal such as a pen to identify the scribe. Also, the distance of the writer may be fixed in advance, or may be measured by providing a distance measuring sensor.

The line-of-sight calculation unit 11 performs coordinate conversion to determine the orientation of the line-of-sight information from the line-of-sight detection camera controller 217 with respect to the screen of the display 280 based on the installation position and orientation information of the line-of-sight detection camera 220 .

Then, the line-of-sight calculation unit 11 calculates the center coordinates of the line-of-sight position on the two-dimensional coordinate plane set on the screen of the display 280 from the direction of the line-of-sight after the coordinate conversion and the position of the writer with respect to the display 280 (hereinafter referred to as "line-of-sight position coordinates”).

The determination unit 12 compares the position coordinates of the line of sight of the writer calculated in real time according to the movement of the writer by the line of sight calculation unit 11 with a predetermined comparison target area, and determines the position coordinates of the line of sight of the writer and the comparison target area. Determine the positional relationship with

The menu display control unit 13 controls the display position of menus and the display of icons.

Here, the line-of-sight calculation unit 11, including the line-of-sight detection camera controller 217, the photographing camera controller 218, and the like, corresponds to "detection means". The determination unit 12 and the menu display control unit 13 correspond to "control means".

(Menu display control method)
Next, a method of controlling display of menus by the control unit 10 will be described. After being activated, the electronic blackboard 2 of the present embodiment displays a menu at the initial setting position on the screen of the electronic blackboard 2 (the screen of the display 280), and starts detecting the line of sight of the writer. When writing on the screen of the electronic blackboard 2, the writer first picks up the electronic pen 290, aligns his/her line of sight with the position on the screen where he/she wishes to write, and places the tip of the pen on the writing position to start writing. . Therefore, if the destination of the line of sight of the writer is within the screen, it can be predicted that the writer is about to start writing. The control unit 10 of the electronic blackboard 2 follows the line of sight of the writer and performs control to move the menu to the vicinity of the writing position of the writer according to the position where the writer intends to write. Also, if the destination of the line of sight of the writer 1 is within the menu, it can be predicted that the writer will select the menu button as the next operation. Therefore, when the writer directs his/her line of sight into the menu to select a menu button, the control unit 10 enlarges the icon of the menu button to which the line of sight is directed and displays an explanation (function name, supplementary explanation, etc.). conduct. Also, when the writer looks away from the screen, it can be predicted that an action other than writing or menu selection will be performed. In this case, the control unit 10 of the electronic blackboard 2 makes the screen easier to see by performing control to return the menu to its original position.

FIG. 3 is a flowchart showing an example of menu display control. 4 and 5 are conceptual diagrams showing an example of the display state of the menu within the screen. FIG. 4 shows a conceptual diagram of movement of the menu area, and FIG. 5 shows a conceptual diagram of enlarged display of icons and the like in the menu area. Below, the flow of the menu display control shown in FIG. 3 will be described with reference to the menu display states shown in FIGS. 4 and 5 as appropriate.

In FIG. 3, after starting line-of-sight detection, the control unit 10 determines whether the writer's line of sight has been detected (step S1). In this example, the control unit 10 detects the line of sight of the writer based on the line of sight information acquired by the line of sight detection camera controller 217 and the photographed image acquired by the photographing camera controller 218 . If the writer is outside the captured image or if the writer does not face the camera (the camera for photography or the camera for detecting the line of sight), the line of sight cannot be detected. In addition, there are cases where an unspecified number of people other than the writer are included in the photographed image or the line-of-sight information, and in such a case, the line of sight of the writer is detected from among the plurality of people. The writer can be identified by, for example, information indicating the position of the writer transmitted from the electronic pen, image analysis of the person holding the electronic pen, the size of the person reflected in the image, and the like. If the line of sight of the writer cannot be detected (step S1: No), detection of the line of sight of the writer is repeated thereafter.

When the line of sight of the writer is detected (step S1: Yes), the control unit 10 then determines whether the position of the line of sight matches the menu area in the screen (step S2).

As shown in FIG. 4, the menu 1000 is displayed at a predetermined position such as the right end of the screen 2800 (the screen of the display 280) by default. Assume that the area (menu area) of the menu 1000 is a rectangular area such as the menu area 1001 shown in FIG. For convenience of explanation, the horizontal axis of the screen 2800 is the X axis, and the vertical axis is the Y axis. . The range in the X-axis direction of the menu area 1001 can be represented by the range of X-coordinates, and the range in the Y-axis direction can be represented by the range of Y-coordinates.

When the line of sight of the writer is detected in step S1, the line of sight position 1002 (coordinates (Xn, Yn)) on the screen 2800 is obtained. ) and the XY coordinate range ((Xp, Yp) to (Xq, Yq)) of the menu area 1001 are compared. Thereby, the control unit 10 can determine whether the obtained line-of-sight position 1002 and the XY coordinate range of the menu area 1001 match. Here, it is assumed that the coordinates indicating the line-of-sight position 1002 and the coordinates on the screen 2800 have the same reference point. Also, in FIG. 4, the line-of-sight position 1002 is indicated by a circle for the sake of explanation, but its center coordinates are (Xn, Yn). Also, (Xn, Yn), (Xp, Yp) to (Xq, Yq) can take arbitrary coordinate points.

If the line of sight does not match the menu area (step 2: No), the control unit 10 subsequently determines whether the direction of the line of sight is within the display (that is, within the screen of the display 280) (step S3). Since the range of the screen 2800 can be indicated by XY coordinates, it is outside the screen 2800 when the line-of-sight position 1002 is outside the range of the screen 2800 .

If the line-of-sight direction is within the screen 2800 (step S3: Yes), the control unit 10 compares the obtained line-of-sight position 1002 with the XY coordinate range of the menu area 1001 to determine whether the distance exceeds a preset threshold. is determined (S4).

If the distance between the line-of-sight position 1002 and the menu area 1001 exceeds the threshold (step 4: Yes), the control section 10 moves the menu to the vicinity of the line-of-sight position (step S5).

As shown in FIG. 4, the control unit 10 moves the menu area 1001 in the direction of an arrow A1 by taking a margin M1, which is the above threshold value, from the line of sight position 1002 of the writer 1. FIG. In this case, the control unit 10 sets the margin M1 in consideration of the reference position of the menu area 1001 (for example, a predetermined reference position such as the center or the corner) and the area width of the menu area 1001. The menu area 1001 is moved to a position that assists writing without interfering with writing. For example, it does not overlap with the line-of-sight position 1002 or the writing position, and moves to a position (near the writing position) where the writer 1 can confirm without greatly deviating the line of sight A2. This is the case when the menu 1000 is in the initial position as in FIG.

After that, the control unit 10 returns to step S1 and performs the same processing. If the writer 1 writes within the range of the margin M1, the distance between the line-of-sight position 1002 and the menu area 1001 is always within the threshold (step 4: No), so the menu area 1001 is arranged at that position. remain. On the other hand, if the writer 1 writes beyond the range of the margin M1, the distance between the line-of-sight position 1002 and the menu area 1001 again exceeds the threshold. The menu area 1001 is brought closer to the line-of-sight position 1002 so that the distance is the margin M1.

After that, it is assumed that the writer 1 looks at the icon of the menu 1000 that has moved near the writing position for editing. The control unit 10 compares the line-of-sight position 1002 with the position of each icon. A description of functions and the like is displayed (step S6).

As shown in FIG. 5, when the line-of-sight position 1002 of the writer 1 matches any of the icons 1000-1 to 1000-3 in the menu area 1001, that icon is enlarged and displayed. FIG. 5 shows a case where the line-of-sight position 1002 coincides with the eraser icon 1000-1. In FIG. 5, an icon description is also displayed on the enlarged icon 1100 . For example, when the writer 1 selects the enlarged icon 1100 (eraser icon), the setting is reflected in subsequent writing functions. In other words, in this example, the eraser function is reflected, and when characters or the like written on the screen are traced from above with the electronic pen, the characters or the like are erased.

After that, the control unit 10 returns to step S1, and, for example, when step S2 results in a No determination, restores the enlarged icon and explanation display to the original display state. In addition, even if step S2 is determined as Yes, if the writer 1 shifts the line of sight to another icon, the enlarged icon and explanation display are returned to the original display state, and the Enlarge the icon and display the description.

When the direction of the line of sight is out of the screen (step S3: No), the control section 10 returns the menu 1000 to the initial position of the menu area 1001 (step S7). This is the case, for example, when the scribe finishes writing and turns his/her eyes away from the screen 2800 or leaves the screen. After step S7, the controller 10 returns to step S1.

In the present embodiment, an example is shown in which line-of-sight detection is started after the electronic blackboard 2 is activated. is received by the electronic blackboard 2, the line-of-sight detection may be started. In that case, since it is possible to stop gaze detection while the electronic pen is not used for writing or menu operations, power consumption can be reduced by turning off some of the devices used for menu display control, thereby saving energy. get the effect of

A program to be executed by the information processing apparatus according to the present embodiment may be provided by being incorporated in the ROM in advance. In addition, it is provided by being recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), etc. as a file in an installable format or an executable format. good too. Alternatively, the software may be stored on a computer connected to a network such as the Internet, and provided by being downloaded via the network.

(Effect of Embodiment)
As described above, in the electronic blackboard shown as an example of the information processing apparatus in the present embodiment, the menu is automatically moved to the writing position where the writer or the like writes, and the menu icon is automatically displayed when selecting the menu. Enlarged display. Therefore, the operability of the writing menu by a writer is greatly improved. In particular, since display control such as movement and enlargement of the menu is performed by detecting the line of sight of the writer, it is possible to move the menu to a position where it is easy to see the menu before the line of sight is diverted to a distant position such as the initial position. has improved dramatically, and you can concentrate on writing. Furthermore, when selecting an icon from the menu, if you reach out to the initial position and select it, the screen will be covered, but since you can select it near your hand, the screen will always be easy for people around you to see. be able to. When the scribe performs an operation other than writing or selecting a menu, the menu returns to its original position, so that there is no portion hidden by the menu and the entire screen can be seen easily.

2 electronic blackboard 10 control unit 11 line-of-sight calculation unit 12 determination unit 13 menu display control unit 201 CPU
202 ROMs
203 RAM
204 SSD
205 Network I/F
206 External device connection I/F
210 Camera 212 GPU
213 display controller 214 contact sensor 215 sensor controller 216 electronic pen controller 217 camera controller for sight line detection 218 photographing camera controller 220 camera for sight line detection 222 power switch 223 selection switches 280 display 290 electronic pen

JP 2017-034502 A

Claims (7)

detection means for detecting the line-of-sight position of the operator on the screen;
When the line-of-sight position detected by the detection means is separated from the position of the menu on the screen, control is performed to bring the menu closer to the line-of-sight position, and the line-of-sight position detected by the detection means is adjusted. a control means for controlling the position of the menu to return to its original position when the screen is out of the screen ;
Information processing device having When the distance between the line-of-sight position detected by the detection unit and the menu on the screen exceeds a threshold, the control means brings the menu closer to the line-of-sight position than the threshold.
The information processing device according to claim 1 . The control means enlarges and displays the menu when the line-of-sight position detected by the detection means matches the position of the menu on the screen.
The information processing apparatus according to claim 1 or 2. The control means displays the explanation of the menu when the line-of-sight position detected by the detection means matches the position of the menu on the screen.
The information processing apparatus according to any one of claims 1 to 3. a transmission means for transmitting a signal operated by an operator;
receiving means for receiving the signal transmitted by the transmitting means;
has
When the signal is transmitted by the transmitting means, the detecting means starts detecting the line-of-sight position .
The information processing apparatus according to any one of claims 1 to 4 . A display control method for controlling display of a menu on a screen, comprising:
a step of detecting the line-of-sight position of the operator on the screen;
when the detected line-of-sight position and the position of the menu on the screen are separated, performing control to bring the menu closer to the line-of-sight position;
a step of controlling the position of the menu to return to its original position when the detected line-of-sight position is out of the screen;
display control methods, including the computer of the information processing device,
detection means for detecting the line-of-sight position of the operator on the screen;
When the line-of-sight position detected by the detection means is separated from the position of the menu on the screen, control is performed to bring the menu closer to the line-of-sight position, and the line-of-sight position detected by the detection means is adjusted. Control means for controlling the position of the menu to return to its original position when the screen is out of the screen ;
A program that acts as a

Images