JP6434594B2 - Image display device, control method for image display device, and image display method - Google Patents

Description

  The present invention relates to an image display device and the like.

  In recent years, portable information processing devices such as smartphones and tablets have become widespread. As the user interface device, a display in which a touch panel is arranged on an image display surface (hereinafter referred to as a touch panel display) is employed. In addition to direct touch operation on the target displayed on the touch panel display, when input of characters and numbers (hereinafter referred to as character input) is required, a software keyboard can be displayed on the screen and operated. Generalized.

  Due to this influence, a touch panel display is also installed in a computer, and a touch operation has been adopted instead of a conventional computer keyboard (hereinafter simply referred to as a keyboard) and a computer mouse. An operating system (hereinafter referred to as an OS) that employs a user interface that assumes a touch operation is also provided.

  For example, in a touch panel display employing an OS capable of displaying an application as shown in FIG. 1, a user can input characters using a software keyboard. Specifically, when a predetermined button 902 in the displayed screen 900 is touched, a software keyboard 912 is displayed as in a screen 910 shown in FIG. Accordingly, as shown in FIG. 3, the user can input characters to the application (window 914) by touching the keys of the software keyboard 912.

  The touch screen display has been increased in screen size, and it is expected that the touch panel display will be used for various purposes in the future. For example, it has been proposed to use an electronic blackboard or the like as a table (such as an office table or a conference table) in which an image display surface is horizontally arranged and can be touched.

  With the spread of such an image display device capable of touch operation, improvement of touch operation is desired. For example, the following Patent Document 1 discloses a keyboard layout displayed on a touch display device based on different finger lengths of users, relative positions between fingers, and habitual operation behavior. A method for adjusting the display appearance of is disclosed. Specifically, by touching the touch display device with the index finger and middle finger of the right hand, the key pitch and key size parameters are calculated in consideration of the size of the display area, and the touch position is changed by dragging the middle finger. According to the above, it is disclosed that a curvature and a curved state of a keyboard are calculated and a curved keyboard is displayed based on the calculated curvature and curved state.

JP 2012-128832 A

  However, a large-screen touch panel display still has a problem with respect to improving operability. For example, an image display apparatus using a large-screen touch panel display with an image display surface arranged horizontally and used as a table has the following problems.

  That is, in order to display the software keyboard, a predetermined button or icon or the like must be selected, and there is a problem that it cannot be displayed immediately.

  In addition, although the position of the user with respect to the display screen is not necessarily constant, the software keyboard is usually arranged at a predetermined position on the screen or a position previously displayed in parallel with the side of the screen. Displayed. Therefore, in the case of a hardware keyboard, it is only necessary to move only the keyboard to a position where key operation is easy, whereas the user moves in accordance with the software keyboard or the user operates the image display device itself. It is necessary to move to an easy position, and the user may feel inconvenient. Even if the user can change the position of the software keyboard, the software keyboard has to be moved to a position where the user can easily use it after the software keyboard is once displayed, which is cumbersome.

  In the first place, unlike a hardware keyboard, there is a problem that a finger cannot be placed in the home position because it is determined as a key input even if a little finger is touched. When entering characters while thinking, sometimes the key operation is interrupted and sometimes the keyboard is touched lightly, and in such a case, it is necessary to release the finger from the keyboard one by one. This may hinder smooth text input.

  These problems cannot be solved by the technique disclosed in Patent Document 1.

  Therefore, the present invention can promptly display the software keyboard at the user's desired position on the image display device when the user wants to input characters, and allows natural key operations without burdening the user. An object of the present invention is to provide an image display apparatus and a control method thereof.

The image display apparatus in the present invention is
Display means for displaying an image;
A detecting means arranged on the image display surface of the display means for detecting a touch position by a user;
Storage means for storing an input device image representing a keyboard that accepts input by a user;
With
The display means includes
When a plurality of touch positions are detected simultaneously by the detection means, the input device image is displayed based on the plurality of touch positions,
When the entire input device image cannot be displayed, the display position of the input device image is moved and displayed based on the plurality of touch positions so that the entire input device image can be displayed.

The image display device of the present invention is
Detecting means for detecting the position of multi-touch on the display screen;
First display means for displaying, on the display screen, an input device image for representing a keyboard that accepts input by a user, in accordance with the position of multi-touch detected by the detection means;
When the detected position of the multi-touch includes a position close to the peripheral edge of the display screen, and the first display unit does not display a part of the input device image, the input device image Second display means for moving and displaying the input device image based on the position of the multi-touch at a position where the entire image is displayed;
It is characterized by providing.

The control method of the image display device of the present invention includes:
A display unit for displaying an image, the arranged display image screen on, a detection unit for detecting a touch position by the user, a storage unit for storing an input device image representing a keyboard for accepting an input by the user A method for controlling an image display device including:
Displaying an image on the display unit;
Detecting a touch position on the image display surface by the detection unit;
When a plurality of touch positions are detected simultaneously by the detection unit, displaying the input device image based on the plurality of touch positions;
When the entire input device image cannot be displayed, the display position of the input device image is moved and displayed based on the plurality of touch positions so that the entire input device image can be displayed;
It is characterized by including.

The image display method of the present invention includes:
A detection step for detecting a multi-touch position on the display screen;
A first display step of displaying an input device image for representing a keyboard for accepting an input by a user on the display screen according to the position of multi-touch detected by the detection step;
When a part of the input device image is not displayed in the first display step because the detected multi-touch position includes a position close to the peripheral edge of the display screen, the input device image is displayed. A second display step of moving and displaying the input device image based on the position of the multi-touch at a position where the entire image is displayed;
It is characterized by including.

  According to the present invention, when a user wants to input via a virtual image simulating an input device on the touch panel display of the image display device, the user simply touches the touch panel in the form of a finger when touching the input device. Thus, the virtual image can be displayed at a position suitable for the user's input operation. Therefore, the user can quickly perform a natural input operation.

  In particular, if the input device is a keyboard for a computer, if the user places his finger on the touch panel so that the user places his finger at the home position of the keyboard, the size corresponds to the size of the user's hand (finger spacing). The software keyboard can be displayed. Therefore, the user can perform a natural key input operation via the displayed software keyboard.

  In addition, when performing key input via the displayed software keyboard, if the user touches four or more points on the software keyboard at the same time, the user does not execute the key input process, so that the user can Key operation can be interrupted with the finger placed on the keyboard, and natural keyboard operation similar to that of a hardware keyboard can be performed.

It is a figure which shows operation for displaying the conventional software keyboard. It is a figure which shows the screen of the state in which the conventional software keyboard was displayed. It is a figure which shows the character input by the conventional software keyboard. It is a block diagram which shows the outline of a structure of the image display apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the detection method of a touch input. It is a figure which shows the state by which the window showing a some application was displayed on the screen. It is a flowchart which shows the control structure of the program for displaying a software keyboard in a user's desired position. It is a figure which shows the state by which the touchscreen display was touched with the several finger | toe. It is a figure for demonstrating the process which determines the position and direction which display a software keyboard. It is a figure which shows a keyboard image. It is a figure which shows the state by which the software keyboard was displayed. It is a figure which shows the state by which the software keyboard was displayed on the position different from FIG. It is a flowchart which shows the control structure of the program for adjusting the size of a software keyboard and displaying it on a user's desired position. It is a figure which shows the state in which the touchscreen display was touched with eight fingers. It is a figure for demonstrating the process which determines the position and direction which display the software keyboard which adjusted the magnitude | size.

  In the following embodiments, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  In the following, “touch” means that the detection device for the input position is in a state in which the position can be detected. When touching and pressing the detection device, when touching without pressing, and without touching In the case of proximity to The input position detection device is not limited to a contact type, and a non-contact type device can also be used. In the case of a non-contact type detection device, “touch” means a state in which the input position is close to the detection device up to a detectable distance.

(First embodiment)
Referring to FIG. 4, image display apparatus 100 according to the first embodiment of the present invention includes an arithmetic processing unit (hereinafter referred to as CPU) 102, a read-only memory (hereinafter referred to as ROM) 104, and rewritable. A memory (hereinafter referred to as RAM) 106, a recording unit 108, a touch detection unit 112, a display unit 114, a display control unit 116, a video memory (hereinafter referred to as VRAM) 118, and a bus 120 are provided. The CPU 102 controls the entire image display apparatus 100.

  As will be described later, the touch detection unit 112 and the display unit 114 constitute a touch panel display, the image display surface of the touch panel display is horizontally arranged, and the image display device 100 is used as a table.

  The ROM 104 is a non-volatile storage device, and stores programs and data necessary for controlling the operation of the image display device 100. The RAM 106 is a volatile storage device from which data is erased when power is turned off. The recording unit 108 is a non-volatile storage device that retains data even when power is cut off, and is, for example, a hard disk drive or a flash memory. The recording unit 108 may be configured to be detachable. The CPU 102 reads a program from the ROM 104 onto the RAM 106 via the bus 120 and executes the program using a part of the RAM 106 as a work area. The CPU 102 controls each part constituting the image display device 100 according to a program stored in the ROM 104.

  A CPU 102, ROM 104, RAM 106, recording unit 108, touch detection unit 112, display control unit 116, and VRAM 118 are connected to the bus 120. Data (including control information) is exchanged between the units via the bus 120.

  The display unit 114 is a display panel (liquid crystal panel or the like) for displaying an image. The display control unit 116 includes a drive unit for driving the display unit 114, reads out image data stored in the VRAM 118 at a predetermined timing, generates a signal for display as an image on the display unit 114, and displays it. Output to the unit 114. The image data to be displayed is read from the recording unit 108 by the CPU 102 and transmitted to the VRAM 118.

  The touch detection unit 112 is a touch panel, for example, and detects a touch operation by the user. The touch detection unit 112 is arranged so as to be superimposed on the display screen of the display unit 114. The touch on the touch detection unit 112 is an operation of designating a point of the image displayed on the display screen corresponding to the touch position. Therefore, in the present specification, in order to eliminate the redundancy of the description, in the case where it is described as a touch on the image displayed on the display unit 114, this description is to the position on the corresponding touch detection unit 112. Means touch. Detection of a touch operation when a touch panel is used as the touch detection unit 112 will be described with reference to FIG.

  FIG. 5 shows an infrared cut-off detection type touch panel (touch detection unit 112). The touch panel is arranged in a row so as to face the LED rows 200 and 202, and the light emitting diode rows (hereinafter referred to as LED rows) 200 and 202 arranged in a row on two adjacent sides of the rectangular writing surface. The two photodiode arrays (hereinafter referred to as PD arrays) 210 and 212 are provided. Infrared light is emitted from the LEDs of the LED rows 200 and 202, and the PDs of the PD rows 210 and 212 facing each other detect the infrared light. In FIG. 5, infrared rays are indicated by upward and leftward arrows from the LEDs of the LED rows 200 and 202.

  The touch panel includes, for example, a microcomputer (hereinafter referred to as a microcomputer) (an element including a CPU, a memory, an input / output circuit, and the like), and controls the light emission of each LED. Each PD outputs a voltage corresponding to the intensity of the received light. The output voltage of the PD is amplified by an amplifier. Further, since signals are simultaneously output from a plurality of PDs in each of the PD columns 210 and 212, the output signals are temporarily stored in a buffer and then output as serial signals according to the arrangement order of the PDs and transmitted to the microcomputer. The The order of the serial signals output from the PD column 210 represents the X coordinate. The order of the serial signals output from the PD column 212 represents the Y coordinate.

  When a user (indicated by a broken line in FIG. 5) 220 touches the touch panel with a finger, infrared rays are blocked at the touched position. Therefore, the output voltage of the PD that has received the infrared light before being cut off decreases. Since the signal portion from the PD corresponding to the touched position (XY coordinate) is decreased, the microcomputer detects the portion where the signal level of the two received serial signals is lowered and obtains the touched position coordinate. . The microcomputer transmits the determined position coordinates to the CPU 102. Since the process of detecting the touch position is repeated at a predetermined detection cycle, if the same point is touched for a longer time than the detection cycle, the same coordinate data is repeatedly output. If it is not touched anywhere, the microcomputer will not transmit the position coordinates.

  Since the touched position detection technique described above is well known, further description will not be repeated. The touch detection unit 112 may be a touch panel (capacitance method, surface acoustic wave method, resistance film method, or the like) other than the infrared blocking method. In the capacitance method, the position can be detected without contact as long as the sensor is close to the sensor.

  The image display apparatus 100 is configured as described above, and the user can operate the image display apparatus 100 in the same manner as a computer. That is, as shown in FIG. 6, the user touches a user interface (such as operation buttons) on the screen 300 displayed on the display unit 114 via the touch detection unit 112 to start an application program. In addition, an operation in a window displayed by the activated application can be performed via the touch detection unit 112. In such a state, when the user inputs characters to a specific application, the software keyboard is displayed. Hereinafter, with reference to FIG. 7, a process for displaying a software keyboard at a position desired by the user will be described.

  The program for displaying the software keyboard on the image display device 100 shown in FIG. 7 is read from, for example, the ROM 104 and executed when the power of the image display device 100 is turned on.

  After the initialization necessary for starting the OS and executing the program is performed, in step 400, the CPU 102 determines whether or not the touch detection unit 112 has been touched. As described above, the CPU 102 determines whether coordinate data has been received from the touch detection unit 112. The touch detection unit 112 does not output the position coordinates unless touched, and outputs the position coordinates (X coordinate, Y coordinate) of the touched point when touched. If it is determined that the touch has been made, the control proceeds to step 402. Otherwise, step 400 is repeated.

  In step 402, the CPU 102 determines whether or not the touch detected in step 400 is a simultaneous touch on a plurality of points of the touch detection unit 112 (hereinafter referred to as “multi-touch”). In the above touch detection method, even when a plurality of points are touched at the same time, the CPU 102 acquires coordinate data of the touch position in time series. For example, the CPU 102 relates to a plurality of coordinate data received within a predetermined time. The distance between the coordinate data and the received coordinate data is calculated, and if the distance is equal to or greater than a predetermined value determined in consideration of the size of the finger, it is determined that a different point is touched, and the distance is less than the predetermined value. If there is, it is determined that the touch is the same point. Therefore, the CPU 102 can determine whether the touch is a single touch or a multi-touch with respect to a plurality of coordinate data received within a predetermined time, and in the case of a multi-touch, the number of points touched simultaneously. And its position can be specified. The touch detection unit 112 determines whether the touch is single touch or multi-touch, and outputs the information and the number of touched points at the same time and the position thereof in the case of multi-touch. May be. If multi-touch, control proceeds to step 404. Otherwise control passes to step 420.

  In step 420, the CPU 102 determines whether or not it is an end operation. The termination instruction is given, for example, by touching a button for terminating the OS on the screen 300. If it is determined that the instruction is to end, the program ends. Otherwise, control passes to step 422.

  In the case of multi-touch, in step 404, the CPU 102 determines whether or not the touch detected in step 400 is multi-touch of four or more points. If it is determined that the number of touches is four or more, the control proceeds to step 406. Otherwise, control passes to step 422.

  If the number of touches is four or more, the CPU 102 determines in step 406 whether the software keyboard is being displayed. For example, when this program is started, a flag area (for example, 1-bit area) is secured at a predetermined address in the RAM 106, and a value indicating that the software keyboard is not displayed, for example, “0” is set as an initial value. In this case, the CPU 102 can determine whether or not the software keyboard is being displayed based on the flag value. If the software keyboard is not being displayed (flag = 0), control proceeds to step 408. Otherwise (if the software keyboard is being displayed), control passes to step 418. The flag is set to a value indicating that the software keyboard is displayed, for example, “1” in step 414 described later.

  In step 418, the CPU 102 determines whether or not four or more multi-touch positions are in the displayed software keyboard. As will be described later, since the CPU 102 displays a software keyboard, the CPU 102 uses information (for example, the coordinates of four vertices) for specifying an area displaying the software keyboard to determine that the multi-touch position is a software keyboard. Whether it is in the keyboard or not can be determined. If it is determined that the software keyboard is touched, control returns to step 400. Otherwise, that is, if an area other than the software keyboard is touched, the control moves to step 422.

  In step 422, the CPU 102 executes processing according to the touch operation detected in step 400. For example, when it is detected that a button displayed on the screen is touched, the CPU 102 executes a process assigned in advance to the touched button. Further, the CPU 102 terminates the application (deletes the window) when the “x” mark in the upper right of each window is touched, and changes the size of the window when the side of the window is touched and dragged. Etc. are executed. For example, when a left-right drag or flick operation is performed on the window of an application that displays a plurality of images in units of pages, the CPU 102 performs a page advance or page return process. When the CPU 102 touches two points on the displayed image at the same time and the distance between the two touched points changes, the CPU 102 executes a process of enlarging or reducing the image, and the other point is placed around one point. When rotated, the image is rotated and displayed. In addition, when a software keyboard is displayed and the software keyboard is touched, key input processing is executed. For example, when three keys of “Ctrl” key, “Alt” key, and “Del” key are touched at the same time, information about the OS (information of the program being executed, etc.) is displayed. Since these touch operations and the processes corresponding thereto are known, they will not be described repeatedly.

  When the software keyboard is not displayed, in step 408, the CPU 102 determines a straight line approximating the two-dimensional distribution of the touched points (hereinafter referred to as an approximate straight line. The approximate straight line is scattered by the multi-touched points. The slope a and the intercept b (the Y coordinate of the point that intersects the Y axis) are stored in the RAM 106. The approximate straight line is represented by y = ax + b. For convenience, the top left vertex of the screen 300 is the origin of coordinates O, the right direction of the screen 300 is the positive direction of the X axis, and the downward direction of the screen 300 is the positive direction of the Y axis. However, the present invention is not limited to this, and how to set the origin of coordinates, the X axis, and the Y axis is arbitrary.

  Here, as illustrated in FIG. 8, it is assumed that the user 600 touches four points on the touch detection unit 112 at the same time using the left and right index fingers and the middle finger. In FIG. 8, the user 600 and the left and right hands of the user are indicated by broken lines. Although the actually detected touch position is a point, the touch position is indicated by white circles 602 to 608 having a predetermined size for convenience.

  In this case, as shown in FIG. 9, the CPU 102 obtains the approximate straight line 302 of the four touched points (white circles 602 to 608) by, for example, a known least square method. In FIG. 9, the window is not displayed for convenience, but the same window as that in FIG. 8 is actually displayed on the screen 300.

  In the least square method, the approximate line is determined so that the sum of values obtained by squaring the vertical distance from each point to the approximate line is minimized. The slope a and the intercept b are obtained by the following formulas 1 and 2. Here, the total number of touch points is n (n ≧ 4), and the coordinates of each touch position i are (xi, yi). 8 and 9 show a case where four points are multi-touched, but the present invention is not limited to this, and any multi-touch of four points or more (n ≧ 4) may be used.

  In step 410, the CPU 102 determines a direction in which the user is facing (hereinafter also referred to as “front direction”). Specifically, the CPU 102 calculates the reciprocal 1 / a of the inclination a stored in the RAM 106, has the reciprocal 1 / a, and is orthogonal to the approximate straight line and the direction of the center of the screen (intersection of diagonal lines) C. Find the vector 304 facing. This vector 304 is used to determine the front direction, the length is arbitrary, and it may be a unit vector.

  In step 412, the CPU 102 calculates the centroid G of the multitouched point. Specifically, the coordinates (xG, yG) of the center of gravity G are obtained by the following equations 3 and 4.

  In step 414, the CPU 102 displays the keyboard image as a software keyboard on the display unit 114, and sets the flag stored in the RAM 106 to a value (for example, “1”) indicating that the software keyboard is being displayed. Here, it is assumed that data of the keyboard image 312 illustrated in FIG. 10 is stored in the recording unit 108 in advance. The CPU 102 reads keyboard image data from the RAM 106 and overwrites it on the image data of the screen 300 on the VRAM 118. When the keyboard image 312 is used by a user as a software keyboard, the upper and lower edges of the keyboard image 312 are such that the upper edge 314 is located farther from the user and the lower edge 316 is closer to the user. To position.

  In FIG. 10, for the sake of convenience, an example of the position of the home position in the key operation is indicated by eight broken circles. These represent the positions where the left and right index finger, middle finger, ring finger, and little finger are placed. The eight dashed circles are not included in the keyboard image 312 itself.

  At this time, the CPU 102 determines the position for displaying the keyboard image 312 as follows. That is, the longitudinal direction (long side) of the keyboard image 312 is parallel to the approximate straight line determined in step 406, and the center of the keyboard image 312 (the intersection of the diagonal lines of the vertices) coincides with the gravity center G. Further, the direction from the lower long side (end 316) to the upper long side (end 314) along the short side of the keyboard image 312 matches the direction of the vector 304 determined in step 408. To. Thus, when the user 600 touches the touch detection unit 112 as shown in FIG. 8, the software keyboard 322 is displayed as shown in FIG.

  When the user's multi-touch position includes a position close to the peripheral edge of the screen, when the keyboard image 312 is displayed so that the center of the keyboard image and the center of gravity G coincide with each other, a part of the keyboard image 312 is not displayed. Sometimes. In that case, it is preferable that the center of the keyboard image 312 be translated along the approximate straight line so that the entire keyboard image 312 is displayed. When the touch position is close to the corner of the screen 300, the entire keyboard image 312 may not be displayed only by translating the keyboard image 312 along the approximate straight line. In that case, it is preferable to further translate the keyboard image 312 in the direction perpendicular to the approximate straight line 302 (the direction of the vector 304).

  In step 416, the CPU 102 activates an application (hereinafter referred to as “key input software”) that accepts key input by the user on the software keyboard 322. Control then returns to step 400.

  The key input software activated here determines which key on the software keyboard 322 the touch position detected by the touch detection unit 112 corresponds to, and generates a code of the determined key (such as an ASCII code). To do. The generated key code is transferred to character generation software such as word processor software, converted into character data, and the character data is transferred to another application in a character input state. As a result, the characters are displayed in the window representing the application in the character input state. In addition, in order to be able to input instructions (system commands) to the OS in addition to character input by key operation, the key code generated by the key input software is used to generate characters such as word processing software via the OS. Configured to be passed to the software. Since the configuration and processing of these software are the same as the processing of known character input software (Japanese input software, etc.) executed on a normal computer, description thereof will not be repeated.

  As described above, the user can execute and stop the application by performing simultaneous touches of three or less points on the screen 300 (whether or not the software keyboard is displayed is arbitrary) displayed on the touch panel display (step). 400 → step 402 → step 420 → step 422, step 400 → step 404 → step 422, or the flow of step 400 → step 402 → step 404 → step 406 → step 418 → step 422).

  On the other hand, when the user 600 does not display the software keyboard on the screen 300 displayed on the touch panel display, the user 600 performs four or more multi-touch on the screen 300, so that the multi-touch is performed on the multi-touched portion. The software keyboard can be displayed in consideration of the position (see the flow in which step 400 to step 416 are sequentially executed). If the user 600 multi-touchs the position not exceeding the center C of the screen 300 as seen from the user 600, the software keyboard 322 displayed on the touch panel display (display unit 114) indicates the position and orientation of the user with respect to the image display device 100. Since it is displayed in consideration, the user 600 can quickly perform natural key operations.

  For example, as shown in FIG. 12, when a user 610 located on the right side of the screen performs multi-touch of four or more points, a software keyboard 332 is displayed. Therefore, when a plurality of users surround the image display apparatus 100 and have a conversation while touching the displayed screen, the software keyboard can be displayed in a direction and a position where each user can easily operate.

  Further, when the user performs multi-touch of four or more points on the software keyboard while the software keyboard is displayed on the screen 300 displayed on the touch panel display, no key input operation is executed. (Refer to the flow of step 400 → step 402 → step 404 → step 406 → step 418 → step 400). Therefore, the user can interrupt the key operation while placing a finger on the software keyboard, and can perform a natural keyboard operation close to the operation feeling of the hardware keyboard.

  Note that the process of deleting the displayed software keyboard is executed in step 422. For example, if the keyboard image 312 shown in FIG. 10 is displayed as a software keyboard, when the “x” key 318 is touched, the CPU 102 erases the software keyboard and ends the application started in step 416. To do. At this time, the CPU 102 sets a flag to “0”.

  In the above, the case where the software keyboard is displayed so that the long side of the software keyboard is parallel to the approximate straight line has been described, but the present invention is not limited to this. The long side of the software keyboard only needs to be along the approximate line, that is, it should be substantially parallel to the approximate line.

  Although the case where an approximate straight line is obtained by the least square method has been described above, the present invention is not limited to this. The approximate line may be determined so that a plurality of multi-touched points are distributed around the approximate line. For example, a straight line connecting two points that are farthest apart may be determined as an approximate straight line.

  Further, an approximate straight line may be obtained by clustering touch points according to the distance. For example, after eliminating points that are far away from other touch points in the set of touch points, an approximate straight line may be obtained using the remaining touch points.

  In the above description, the case where the software keyboard is displayed so that the center of the software keyboard (keyboard image 312) coincides with the center of gravity G obtained in step 412 has been described, but the present invention is not limited to this. Since the keyboard is not always symmetrical, the software keyboard may be displayed so that a predetermined representative point of the keyboard image matches the center of gravity G obtained in step 412.

  Although the case where the software keyboard is displayed by multi-touch of four or more points has been described above, the present invention is not limited to this. If the keyboard is not displayed, the software keyboard may be displayed by multi-touch of three or more points. In addition, a software keyboard may be displayed when two or more points are touched simultaneously in an area other than an area representing some object such as a window, icon, or button representing an application. When displaying a software keyboard with two points of multi-touch, in order to determine the direction of the software keyboard, a straight line passing through the two touched points may be used instead of the approximate straight line. The process for obtaining the approximate straight line is unnecessary.

  Although the case where the software keyboard shown in FIG. 10 is displayed has been described above, the present invention is not limited to this. The type of software keyboard to be displayed (key layout, key shape, displayed language, etc.) is arbitrary. Further, the shape of the software keyboard is not limited to a rectangle, and is arbitrary. Moreover, it is not limited to the software keyboard for computers. It is a virtual input image representing an input device displayed on the display unit 114. When the user performs a touch operation with a finger, the difficulty level of the input operation varies depending on the arrangement (particularly the orientation) of the input image with respect to the user. The present invention can be applied to such an input image. For example, it may be a music keyboard (keyboard) for inputting notes to a composition application. Further, the keyboard image stored in the recording unit 108 is not limited to bitmap data. The keyboard image stored in the recording unit 108 includes CG data that can generate a bitmap keyboard image by computer graphics (CG).

(Second Embodiment)
In the first embodiment, a case has been described in which a keyboard image prepared in advance is displayed in the same size, but in the second embodiment, the keyboard size is changed according to the touch position of the user. Change and display.

  The image display apparatus according to the second embodiment has the same configuration as the image display apparatus 100 described as the first embodiment and has the same function. In the following, the repeated description will not be repeated, and points different from the first embodiment will be mainly described.

  In the image display apparatus 100 shown in FIG. 13, a program for adjusting and displaying the size of the software keyboard is started up and executed in the same manner as the program shown in FIG. FIG. 13 differs from FIG. 7 only in that step 500 to step 504 are added and step 414 is changed to step 506. As shown in FIG. 14, the added step is executed when the user touches eight points on the touch detection unit 112 with fingers other than the left and right thumbs at the same time. In FIG. 13, at the step given the same number as in FIG. 7, the CPU 102 executes the same process as the step in FIG. 7.

  After starting the program, in step 400, the CPU 102 determines whether or not the touch detection unit 112 has been touched. As shown in FIG. 14, if the user performs 8 multi-touch, the processes in steps 400 to 412 are executed in order.

  In step 500 following step 412, the CPU 102 determines whether or not the multi-touch is 8 points. If it is determined that the multi-touch is 8 points, the control proceeds to step 502. Otherwise, control passes to step 414.

  In step 502, the CPU 102 determines a rectangular area including all multi-touch points. Specifically, as shown in FIG. 15, the CPU 102 has a rectangular area 706 surrounded by two straight lines parallel to the approximate straight line 702 obtained in step 408 and two straight lines perpendicular to the approximate straight line 702. As four pieces of information, the four straight lines are determined so as to include all the multi-touch points and the area of the rectangular area is minimized, the coordinates of the four intersections of the straight lines are calculated, and the rectangular area 706 is specified. Store in the RAM 106.

  In step 504, the CPU 102 determines the size of the software keyboard from the information (coordinates of the four vertices) specifying the rectangular area 706 stored in the RAM 106. Here, it is assumed that the reference length L of the keyboard image 312 is stored in the recording unit 108 in addition to the data of the keyboard image 312 shown in FIG. The reference length L is, for example, the center-to-center distance (pixel value) of two circles at the left and right ends (circle representing the touch position of the left and right little fingers) among the eight broken circles.

  The CPU 102 reads the reference length L from the recording unit 108, divides the long side length of the rectangular area 706 by the reference length L, and calculates the ratio of the long side length of the rectangular area 706 to the reference length L.

  In step 506, the CPU 102 reads the data of the keyboard image 312 from the recording unit 108, and displays the data by scaling (enlarging or reducing) the reference length L to be equal to the length of the long side of the rectangular area 706. Specifically, the CPU 102 generates image data obtained by scaling the keyboard image 312 (hereinafter referred to as “display keyboard image”) at the ratio calculated in step 504, and stores it in the RAM 106. Further, the CPU 102 reads display keyboard image data from the RAM 106 and overwrites the image data corresponding to the screen 300 on the VRAM 118. Thus, the display keyboard image is displayed on the display unit 114 as a software keyboard.

  At this time, the CPU 102 determines the position for displaying the display keyboard image as follows. That is, the longitudinal direction (long side) of the display keyboard image is parallel to the approximate straight line determined in step 406, and the center of the display keyboard image (the intersection of the diagonals of the vertices) is made to coincide with the center of gravity G. . Furthermore, the direction from the lower long side (end 316) toward the upper long side (end 314) along the short side of the display keyboard image matches the direction of the vector 704 determined in step 408. Like that. Thus, when the user touches the touch detection unit 112 as shown in FIG. 15, the display keyboard image is displayed as a software keyboard as in FIG. However, the software keyboard displayed at this time is a display keyboard image whose size has been adjusted.

  As described above, in the second embodiment, when the user performs a touch operation other than the eight simultaneous touches on the screen 700, the same processing as in the first embodiment can be performed. In addition, when the user performs 8 simultaneous touches on the screen 700 on which the software keyboard is not displayed, the size of the keyboard image 312 is adjusted according to the relative distance of the touch points. Can be displayed. For example, a software keyboard having a size corresponding to the size of the user's hand can be displayed, and a software keyboard having an appropriate key pitch (adjacent key interval) can be displayed to the user. Therefore, user key operations can be made easier.

(Third embodiment)
In the second embodiment, a case has been described in which information specifying the rectangular area 706 including eight touch points is obtained, but the present invention is not limited to this. In the third embodiment, as a distance corresponding to the reference length L, a distance between two points having a maximum relative distance among eight touch points is obtained.
The image display device according to the third embodiment has the same configuration as the image display device 100 according to the first embodiment, and has the same function. Unlike the second embodiment, the program for adjusting and displaying the size of the software keyboard executed in the image display device according to the third embodiment differs from that in the second embodiment in FIG. The process of determining the display size in step 504 is executed without the process of determining). In step 504, the CPU 102 obtains a distance Lmax between two points having the maximum relative distance among the eight touched points, reads the reference length L of the keyboard image 312 from the recording unit 108, and sets the distance Lmax as the reference length L. Divide to calculate the ratio Lmax / L. After that, as in the second embodiment, in step 506, the CPU 102 reads the keyboard image 312 data from the recording unit 108 and scales it using the ratio Lmax / L to generate a display keyboard image. Based on the approximate straight line 702 and the center of gravity G, a display keyboard image is displayed.
As described above, in the third embodiment, when the user performs a touch operation other than the eight touches simultaneously on the screen 700, the same processing as in the first embodiment can be performed. In addition to this, when the user performs eight simultaneous touches on the screen 700 on which the software keyboard is not displayed, the size of the keyboard image 312 can be adjusted and displayed.

(Fourth embodiment)
In the third embodiment, a case has been described in which the size of the software keyboard is adjusted according to the distance between two points having the maximum relative distance among the eight touch points, that is, the distance between the touched little fingers. However, it is not limited to this. In the fourth embodiment, a predetermined reference length obtained from the position of the home position of the keyboard image 312 is stored in the recording unit 108, and a length corresponding to the reference length is calculated from the position of the touch point. The keyboard image 312 is scaled using the ratio of the obtained value to the reference length.
The image display device according to the fourth embodiment has the same configuration as the image display device 100 according to the first embodiment, and has the same function. In the program for adjusting and displaying the size of the software keyboard executed in the image display apparatus according to the fourth embodiment, as in the third embodiment, step 502 (rectangular The process of determining the display size in step 504 is executed without the process of determining). Specifically, the keyboard image so that the two innermost points (corresponding to the touch points of the left and right index fingers) of the eight touch points coincide with the “f” key and the “j” key of the software keyboard. 312 is scaled and displayed. The distance D1 between the “f” key and the “j” key of the keyboard image 312 is stored in the recording unit 108 in advance. In step 504, the CPU 102 calculates the distance D2 between the two innermost points of the eight touch points, reads the distance D1 from the recording unit 108, and calculates the ratio D2 / D1. Thereafter, as in the second embodiment, in step 506, the CPU 102 scales the keyboard image 312 read from the recording unit 108 by the ratio D2 / D1 to generate a display keyboard image. Then, as described above, the display keyboard image is displayed based on the approximate straight line 702 and the center of gravity G.
As described above, in the fourth embodiment, when the user performs a touch operation other than the eight simultaneous touches on the screen 700, the same processing as that in the first embodiment can be performed. In addition to this, when the user performs eight simultaneous touches on the screen 700 on which the software keyboard is not displayed, the size of the keyboard image 312 can be adjusted and displayed.
The key for setting the reference length may be a key other than the “f” key and the “j” key. For example, “d” key and “k” key, “s” key and “l” key, or other key combinations may be used.

(Fifth embodiment)
In the fifth embodiment, among the plurality of points representing the home position of the keyboard image 312, at least two positions are stored as reference point positions in the recording unit 108 and correspond to the two reference points. The keyboard image 312 is scaled so that the positions of the two touch points coincide with the positions of the two reference points.
The image display apparatus according to the fifth embodiment has the same configuration as the image display apparatus 100 according to the first embodiment, and has the same function. In the program for adjusting and displaying the size of the software keyboard executed in the image display apparatus according to the fifth embodiment, in FIG. 13, step 412 (processing for calculating the center of gravity), step 502 (rectangular Step 506 is executed without performing the determination process) and step 504 (display size determination process). In step 506, the CPU 102 determines that the position of the center of each of the “f” key and the “j” key of the keyboard image 312 matches the two innermost points of the eight touch points. Is scaled and rotated for display. In that case, the process for obtaining the center of gravity G as described above is not necessary. In order to determine the front direction of the user, a process for obtaining an approximate straight line is necessary.
As described above, in the fifth embodiment, when the user performs a touch operation other than the eight simultaneous touches on the screen 700, the same processing as that in the first embodiment can be performed. In addition to this, when the user performs eight simultaneous touches on the screen 700 on which the software keyboard is not displayed, the size of the keyboard image 312 can be adjusted and displayed.
It should be noted that keys of home positions other than the “f” key and the “j” key may be used. For example, “d” key and “k” key, “s” key and “l” key, or other key combinations may be used.

  The present invention has been described above by describing the embodiment. However, the above-described embodiment is an exemplification, and the present invention is not limited to the above-described embodiment, and is implemented with various modifications. be able to.

  The image display device according to the above-described embodiment is designated by an operation for designating a display unit for displaying an image, an image display surface of the display unit, and a position on the image displayed by the display unit. A detection unit for detecting a position to be detected, a straight line calculation unit for calculating a straight line approximating a two-dimensional distribution of the plurality of detected positions in response to the simultaneous detection by the detection unit, and a touch by the user And a storage unit that stores an image representing an input device that is an operation target. The display unit displays the image read from the storage unit as a virtual image representing the input device. When the input device is used by a user, the input device includes two ends facing the user, and an image representing the input device is displayed at an end of the input device that is located farther from the user. A corresponding first end and a second end corresponding to the end located closer to the user. In the image displayed as the virtual image, the horizontal direction of the image corresponding to the horizontal direction of the input device when the input device is used by the user is along an approximate straight line, and the first end and the second end are The first end portion may be located closer to the center of the image display surface than the second end portion.

  Preferably, the image display device further includes a centroid calculation unit that calculates centroids of a plurality of positions detected by the detection unit, and the display unit further includes a representative point of an image representing the input device so as to match the centroid. A virtual image may be displayed.

  More preferably, the input device is a computer keyboard, and the storage unit further stores a reference length obtained from the position of the home position on the image representing the input device. The image display device includes a distance calculation unit that calculates a distance corresponding to a reference length obtained from the detected eight points in response to detection of eight positions by the detection unit, and a distance calculation unit for the reference length And an image processing unit that generates an image for display by scaling an image representing the input device at the distance ratio calculated by the above. The display unit may display a display image instead of the image representing the input device as a virtual image representing the input device in response to detection of the positions of the eight points by the detection unit.

  More preferably, the virtual image is an image of a computer keyboard, and the storage unit further stores at least two positions among eight points representing the home position on the image representing the input device. In response to the detection of the positions of the eight points by the detection unit, the image display device identifies two of the eight points corresponding to two points representing the home position stored in the storage unit And generating the display image by scaling the image representing the input device so that the positions of the two points identified by the point identification unit coincide with the two points representing the corresponding home position stored in the storage unit. An image processing unit. The display unit may display a display image instead of the image representing the input device as a virtual image representing the input device in response to detection of the positions of the eight points by the detection unit.

  Preferably, in a state where the virtual image is displayed on the display unit, the image display device performs input processing corresponding to the detection in response to the detection unit detecting that less than a predetermined number of positions on the virtual image are simultaneously detected. An input unit to be executed is further provided. The input unit may not perform the input process when the detection unit simultaneously detects a predetermined number or more of the positions of the virtual images.

  More preferably, the predetermined number may be four.

  The control method of the image display apparatus according to the above-described embodiment is designated by an operation for designating a position on the image displayed on the image display surface of the display unit and the display unit that displays the image and displayed on the display unit. It is a control method of an image display device including a detection unit that detects a position and a storage unit that stores an image representing an input device that is a target of a touch operation by a user. The control method includes a step of displaying an image on the display unit, a detection step of detecting a position specified by an operation of specifying a position on the image displayed on the display unit by the detection unit, and a plurality of detection steps. An image read from the storage unit as a virtual image representing the input device on the display unit, and a step of calculating a straight line approximating the two-dimensional distribution of the detected positions of the plurality of points in response to the simultaneous detection of the positions Displaying. When the input device is used by a user, the input device includes two ends facing the user, and an image representing the input device is displayed at an end of the input device that is located farther from the user. A corresponding first end and a second end corresponding to the end located closer to the user. In the image displayed as the virtual image, the horizontal direction of the image corresponding to the horizontal direction of the input device when the input device is used by the user is along an approximate straight line, and the first end and the second end are The first end portion may be located closer to the center of the image display surface than the second end portion.

100 Image Display Device 102 Arithmetic Processing Unit (CPU)
104 Read-only memory (ROM)
106 Rewritable memory (RAM)
108 Recording unit 112 Touch detection unit 114 Display unit 116 Display control unit 118 Video memory (VRAM)
120 bus

Claims (9)

Display means for displaying an image;
A detecting means arranged on the image display surface of the display means for detecting a touch position by a user;
Storage means for storing an input device image representing a keyboard that accepts input by a user;
With
The display means includes
When a plurality of touch positions are detected simultaneously by the detection means, the input device image is displayed based on the plurality of touch positions,
When the entire input device image cannot be displayed, the display position of the input device image is moved and displayed based on the plurality of touch positions so that the entire input device image can be displayed. .   The display means translates the center of the input device image along a straight line approximating the two-dimensional distribution of the plurality of touch positions, and translates the center of the input device image along the straight line. When the entire input device image cannot be displayed, the input device image is moved and displayed by translating the center of the input device image along a direction perpendicular to the straight line. Item 4. The image display device according to Item 1. The input device image includes a first end and a second end,
When a plurality of touch positions are simultaneously detected by the detection unit, the display unit is on the same side with respect to the center of the image display surface, and the first end is more than the second end. The image display device according to claim 1, wherein the input device image is displayed at a position closer to the center of the image display surface.   When the plurality of touch positions include a position close to a peripheral edge of the screen display surface, the display means sets the display position of the input device image so that the entire input device image can be displayed. The image display device according to claim 1, wherein the image display device moves and displays based on a touch position.   The said display means displays the said input device image so that the representative point of the said input device image may correspond to the gravity center based on these several touch positions, The said input device image is characterized by the above-mentioned. The image display device described in 1. The storage means further stores a reference length obtained from a position of a home position on the input device image;
In response to the detection of a plurality of touch positions by the detection means, distance calculation means for calculating a distance corresponding to the reference length obtained from the detected plurality of touch positions;
Image processing means for generating a display image by scaling the input device image at a ratio of the distance calculated by the distance calculating means to the reference length;
Further comprising
The display means displays the display image instead of the input device image as the input device image in response to detection of a plurality of touch positions by the detecting means. The image display device according to any one of 1 to 5. Detecting means for detecting the position of multi-touch on the display screen;
First display means for displaying, on the display screen, an input device image for representing a keyboard that accepts input by a user, in accordance with the position of multi-touch detected by the detection means;
When the detected position of the multi-touch includes a position close to the peripheral edge of the display screen, and the first display unit does not display a part of the input device image, the input device image Second display means for moving and displaying the input device image based on the position of the multi-touch at a position where the entire image is displayed;
An image display device comprising: A display unit for displaying an image; a detection unit which is arranged on an image display surface of the display unit and detects a touch position by a user; and a storage unit which stores an input device image representing a keyboard for receiving an input by the user; A method for controlling an image display device including:
Displaying an image on the display unit;
Detecting a touch position on the image display surface by the detection unit;
When a plurality of touch positions are detected simultaneously by the detection unit, displaying the input device image based on the plurality of touch positions;
When the entire input device image cannot be displayed, the display position of the input device image is moved and displayed based on the plurality of touch positions so that the entire input device image can be displayed;
The control method characterized by including. A detection step for detecting a multi-touch position on the display screen;
A first display step of displaying an input device image for representing a keyboard for accepting an input by a user on the display screen according to the position of multi-touch detected by the detection step;
When a part of the input device image is not displayed in the first display step because the detected multi-touch position includes a position close to the peripheral edge of the display screen, the input device image is displayed. A second display step of moving and displaying the input device image based on the position of the multi-touch at a position where the entire image is displayed;
An image display method comprising:

Images