JP2015088090A - Operation display device, and operation display method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a plurality of users to comfortably perform touch operations at the same time on a display device on which a touch operation can be done.SOLUTION: A table display (10) is configured to display an operation pad, enabling a display object displayed at a display part (21) to be operated by a touch, at the display part (21). The table display includes: an operation detection part (51) which detects a touch operation; an operation determination part (52) which determines an operation object and operation details of the touch operation from the detection result; an operation execution part (54) which generates image data on an operation pad where a display object is an operation object and generates image data for execution of an operation on the display object by a touch operation on the operation pad when it is determined that a pad display operation to display the operation pad of the display object is done; and a display control part (30) which allows the display part 21 to make a display based upon the image data.

Description

  The present invention relates to an operation display device, an operation display method, and a program for performing a touch operation on a display provided with a touch panel.

  Conventionally, when operating a screen with a touch-compatible display device such as a large table display, it is necessary to move the hand greatly when the screen is large, so that the touch cannot be performed comfortably. Further, when a plurality of people perform operations separately, when the operation target is moved by a touch operation, for example, hands may cross each other, and the operability is poor. Therefore, it was necessary to prepare a separate input device such as a mouse and a terminal for operation.

  In order to solve this, Patent Document 1 discloses that a virtual touchpad is displayed on a display screen and operated by the touchpad. Patent Document 2 discloses that a body detection unit detects a user's position and the like, and displays a control window (touch pad) at a position where the user can easily operate. Japanese Patent Application Laid-Open No. 2004-151620 discloses a method in which a plurality of user's surrounding operations are provided on a table display, and a shared area is provided to copy or move from each user's dedicated area via the shared area.

JP 2013-15890 A JP 2012-103800 A JP 2005-352579 A

  However, the technique of Patent Document 1 enlarges the operation area of the touchpad so that it can be easily operated, but does not assume simultaneous operation by a plurality of users. Therefore, when a plurality of users operate from multiple directions at the same time, the operability is not improved.

  Moreover, in the technique of patent document 2, while assuming that it is used by a plurality of users, the touch pad is displayed at a position that is easy for each user to use, but the display device is a vertical display, Only operation from one direction around the display is assumed. Therefore, when it is operated from each direction around the display like a table display, the operability is not improved.

  The technique of Patent Document 3 is operated by a plurality of users from each direction around the table display and performs copying and moving through the common area. However, depending on the position of the common area, the operation distance is large, Sex is not improved.

  When multiple people touch and operate windows and objects displayed on a large-screen display at the same time, it may be confusing, and eventually it will be time consuming and efficient for each user to operate. I couldn't say that.

  In view of such circumstances, the present invention is intended to provide an operation display device, an operation display method, and a program that can be comfortably operated when a plurality of users perform a touch operation simultaneously on a display device that can be touch-operated. is there.

The present invention is an operation display device for displaying on the display unit an operation pad that enables an operation by touching the display object displayed on the display unit,
An operation detection unit that detects a touch operation, an operation determination unit that determines an operation target and an operation content in the touch operation based on a detection result of the operation detection unit, and the operation determination unit that displays the operation pad of the display object. When it is determined that a pad display operation has been performed, image data of the operation pad having the display object as an operation object is generated, and an operation to the display object is performed by a touch operation on the operation pad. An operation execution unit for generating image data;
And a display control unit that displays on the display unit based on the image data.

  Here, the display object is an application window or an image object.

  Further, the operation execution unit displays the operation pad in the vicinity of the screen periphery of the display unit closest to the display object.

  The operation execution unit may be provided with a button for operating itself on the operation pad, and a touch operation on the operation pad itself may be performed by a button operation on the operation pad.

  In addition, when the pad display operation is performed on a portion other than the display object, the operation determination unit determines that the operation pad targets a cursor movement operation displayed on the display unit. It is characterized by.

  In addition, when the operation content is determined to be a character input mode by the operation determination unit, the operation determination unit further includes a character recognition unit that recognizes a character manually input to the operation pad, and the operation execution unit includes the character recognition unit The character recognized by the above is generated as a character image.

Further, the present invention is an operation display method for displaying on the display unit an operation pad that enables an operation by touching the display object displayed on the display unit,
An operation detection step for detecting a touch operation, an operation determination step for determining an operation target and operation content in the touch operation based on a detection result of the operation detection step, and the operation pad of the display object is displayed by the operation determination step. When it is determined that a pad display operation has been performed, image data of the operation pad having the display object as an operation object is generated, and an operation to the display object is performed by a touch operation on the operation pad. An operation execution step for generating image data and a display control step for displaying on the display unit based on the image data are provided.

  Moreover, this invention is a program for making a computer perform each step of the said operation display method.

  According to the present invention, when a pad display operation for displaying an operation pad whose operation target is a display object (an application window or an image object) is performed, the operation pad is displayed and touched with the operation pad. Since the operation on the display object is performed by the operation, the user can operate the display object only by touching the operation pad. In particular, since the operation pad corresponding to the display object to be operated is displayed, the user can easily recognize and operate. When the display object is moved greatly, it is only necessary to operate the operation pad, and even a plurality of users can perform simultaneous operation without confusion, so that the operation efficiency is improved.

It is an external view which shows the table display which is the operation display apparatus which concerns on this invention. It is explanatory drawing in which a user operates a display target object with a table display. It is explanatory drawing which shows an example of an operation pad. It is a functional block diagram which shows an example of a table display. It is a flowchart which shows the operation process with respect to the operation target in 1st Embodiment. It is explanatory drawing which shows movement operation of an operation pad. It is explanatory drawing which shows movement operation of a display target object. It is a flowchart which shows the operation process with respect to the handwritten input in 2nd Embodiment. It is explanatory drawing which shows the handwritten character input by an operation pad.

Embodiments of the present invention will be described below with reference to the accompanying drawings.
FIG. 1 is an external view showing a table display which is an operation display device according to the present invention. FIG. 2 is an explanatory diagram in which the user operates the display object on the table display. FIG. 3 is an explanatory diagram illustrating an example of the operation pad.

<Overall configuration and operation outline>
As shown in FIGS. 1 and 2, the table display 10 is a device that includes a display device 20 that is a touch panel on the entire surface of the table, and a plurality of users perform touch operations while watching the display to perform processing. The display device 20 has a structure including a touch input unit on the upper side of a display unit such as a liquid crystal panel.

  As shown in FIG. 2, the user who surrounds the table display 10 is displayed on the application windows W1 and W2 and the image objects O1 and O2 displayed on the display device 20 with a movement operation in the arrow direction. The operation pad 100 (100-1, 100-2, 100-3, 1004) is used. If each user moves directly to the application window W1, W2 and the objects O1, O2, which are the display objects, by touch operation, if the moving distance is long, he / she has to get out, The operability is not good because the arms cross with other users. Therefore, the operation pad 100 (100-1, 100-2, 100-3, 1004) is displayed in the vicinity of each user, and the application windows W1, W2 and the display objects O1, O2 are moved by touch operation. I do. The operation pads 100 (100-1, 100-2, 100-3, 1004) correspond to the application windows W1, W2 and the objects O1, O2 on a one-to-one basis. Further, the operation pad 100 is used to operate the cursor when there is no operation target (when no operation target is specified).

<Appearance structure of display operation pad>
As shown in FIG. 3, an operation button for the operation pad 100 itself is displayed at the left end of the operation pad 100. A “pen soft remote control” button 101 for handwriting input, a “move” button 102 for moving, a “magnification” button 103 for changing display magnification, a “rotation” button 104 for rotating, and an operation display pad A “close” button 105 for closing is provided.

  In addition, buttons for operating display objects (windows and objects) displayed on the display device 20 are displayed above and below the operation pad 100. On the upper side, a reduction button 106 that reduces the selected display object at a certain rate, a maximize button 107 that maximizes the selected display object, and an enlargement button 108 that enlarges the selected display object at a certain rate. Is provided. On the lower side, a previous page button 109 for returning the page of the selected window and a next page button 110 for sending the page of the selected window are provided.

A touch pad portion (portion surrounded by a dotted line) 111 of the operation pad 100 is a portion that performs touch operations such as tap, flick, and pinch.
Although there is no button for moving and rotating the display object, for example, the following operation is performed. In order to move, two places on the touch pad unit 111 are simultaneously touched and one of them is slid in the moving direction. To rotate, touch two locations at the same time and rotate one in the direction of rotation.
Note that this is merely an example of an operation on a display object, and a movement button or a rotation button may be provided.

  In addition, the operation pad 100 can perform a gesture operation by a finger operation, and can move, scale, rotate, and maximize a display object. For example, by twisting two touched fingers. Various gesture operations for manipulating the rotation direction and angle may be set. The presenter can operate the smart touchpad to change the display data to an optimal state, thereby enabling an optimal presentation to the viewer.

  In order to display the operation pad 100, for example, the display operation of the operation pad 100 is to press and hold the display screen of the display device 20 for a predetermined time (for example, 5 seconds) or longer. In this case, when a display target object (window or object) to be operated is pressed for a long time, the display target object is designated as the operation target. In this way, the operation pad 100 corresponds to the display object on a one-to-one basis.

  Further, if the long-pressed position is a place where there is no display object, the operation pad 100 performs a cursor movement process. In this case, since no operation is performed on the window or the object, the display object operation button is displayed lightly to notify the user that it cannot be used. Or in order to show cursor operation, you may display the image which shows a cursor also in the corner of an operation pad.

  In the following embodiments, the operation pad 100 is made to appear by long-pressing the display screen. However, the touch position such as making the operation pad 100 appear by touching the same portion two times can be specified. It may be an operation.

<Functional block configuration>
FIG. 4 is a functional block diagram illustrating an example of a table display.

  The table display 10 includes a display device 20 that is a touch panel, a display control unit 30 that performs display control on the display device 20, a storage unit 40 that stores data, and a control unit 50 that controls processing according to an operation.

  As described above, the display device 20 has a structure including the touch input unit 22 on the upper side of the display unit 21 such as a liquid crystal panel.

  The control unit 50 includes an operation detection unit 51 that detects the touch position and touch movement from the input signal of the touch input unit 22, an operation determination unit 52 that determines an operation target and operation content from the detected touch position and movement, and the determined touch. An operation execution unit 53 that executes an operation process based on an operation to generate an image and a character recognition unit 54 that performs character recognition from a signal from the touch input unit 22 are provided.

  The display control unit 30 performs display on the display unit 21 based on the image data generated by the control unit 50.

  The storage unit 40 is a storage device including a hard disk, a memory, and the like, stores image data and various types of data related to operation processing, and exchanges data with the control unit 50.

[First Embodiment]
The first embodiment relates to an operation on an operation target using the operation pad 100. Handwriting input by the pen software remote control button 101 will be described later as a second embodiment.

FIG. 5 is a flowchart showing an operation process for the operation target in the first embodiment.
The user touches the screen of the display unit 21 (step S1). At this time, when operating the display object (window or object) displayed on the display unit 21, the user touches the window or object to be operated. When operating the cursor, a place where there is no operation target, that is, a place where no window or object is displayed is touched.

  The operation determination unit 52 determines whether the touch operation has been pressed for a predetermined time or longer (step S2). The predetermined time is set in advance and stored in the storage unit 40, and the operation determination unit 52 reads the predetermined time from the storage unit 40 and uses it for determination. If the button has been pressed for a predetermined time or longer, the operation determination unit 52 determines that an instruction to display the operation pad 100 has been issued. The operation execution unit 53 generates image data for displaying the operation pad. The display control unit 30 displays the operation pad 100 on the display unit 21 based on the image data (step S3).

  At this time, the operation determination unit 52 determines which window or object the user is operating from the location where the user is long-pressed, and stores the data in the storage unit 40.

In addition, the display position of the operation pad 100 is set in advance, such as a long-pressed position, a vicinity of the long-pressed position, or a display target object (window or object) to be operated. As shown in FIG. 2, it may be displayed near the periphery of the display screen that is closest to the display object that is the operation target. In the table display, since there is a user along the periphery of the display screen and the user can move along the periphery of the display screen, if the operation pad 100 is displayed in the vicinity of the peripheral portion, the user can easily operate. Become.
The display position of the operation pad 100 may be arbitrarily set by the user.

  The user touches the operation pad 100 (step S4). The operation determination unit 52 determines whether the operation button 101 to 105 of the operation pad 100 is a touch from the detection result of the operation detection unit 51 (step S5). If it is a touch on the operation buttons 101 to 105, the operation target is for the operation pad 100 itself. Therefore, the operation process is performed on the operation pad 100 according to the operation process indicated by the touched operation button (step S6). ). That is, the operation determination unit 52 determines the operation content based on the detection result of the operation detection unit 51, and sends the determination result to the operation execution unit 53. The operation execution unit 53 moves and rotates the operation pad 100 based on the content of the operation, and generates the image data. The display control unit 30 displays the operation pad 100 on the display unit 21 based on the image data.

When the movement button 102 is touched, the operation pad 100 is moved in that direction by a slide operation or a flick operation on the touch pad unit 111.
When the scaling button 102 is touched, the operation pad 100 is enlarged or reduced in that direction by a pinch operation on the touch pad unit 111.
When the rotation button 104 is touched, the operation pad 100 is rotated in the rotation direction by a rotation slide operation on the touch pad unit 111.
When the close button 105 is touched, the operation pad 100 is closed (the image display is stopped).

  Regarding movement and scaling, the touch operation of the operation pad 100 and the degree of movement and scaling on the screen of the display unit 21 are set in advance, and the user can arbitrarily set them.

A specific example of the operation process of the operation pad 100 will be described. FIG. 6 is an explanatory diagram showing the operation of moving the operation pad.
When the user touches the “move” button 102 of the operation pad 100, the black and white display of the button 102 is reversed to indicate that the movement processing mode has been entered. When the user performs a slide operation on the touch pad unit 111 of the operation pad 100, the operation pad 100 is moved and displayed in the slide direction on the screen of the display unit 21.

  When the display object is maximized on the display device 20, the maximized display object is positioned at the foreground, the other display objects and the operation pad 100 are positioned at the back, It is convenient as a presentation. However, in the case of such a display, since the operation pad 100 corresponding to the maximized display object is also located on the back side, it is maximized by directly touching the maximized display object. You can return to the previous state.

  Next, when the operation target is not the operation pad 100 in step S5, it is determined whether the operation target is a display target (window or object) (step S7). Since the operation target is stored in the storage unit 40 when the operation pad 100 is displayed, the operation determination unit 52 determines based on the operation target data. If the operation target is a display target, the operation is performed on the display target as the operation target based on the operation performed on the operation pad 100 (step S8). That is, the operation determination unit 52 determines the operation content based on the detection result of the operation detection unit 51, and sends the determination result to the operation execution unit 53. The operation execution unit 53 moves and rotates the display target object (window or object) to be operated based on the content of the operation, and generates the image data. The display control unit 30 displays on the display unit 21 the display object that has been operated based on the image data.

The operation process for the display object (window or object) is as follows.
When the reduction button 106 is touched, the image is reduced at a certain rate.
When the maximize button 107 is touched, the screen of the display unit 21 is enlarged to the maximum.
When the enlarge button 108 is touched, the image is enlarged at a certain rate.
When the previous page button 109 is touched, the page of the operation target window is returned.
When the next page button 110 is touched, the window page to be operated is sent.
When moving the display object, the touch pad unit 111 touches two places at the same time, and slides one touch in the moving direction.
When rotating the display object, two places are touched simultaneously on the touch pad unit 111, and one touch is rotated and slid in the rotation direction.

  Regarding the movement and scaling, the touch operation of the operation pad 100 and the degree of movement and scaling of the display object on the screen of the display unit 21 are set in advance, and can be arbitrarily set by the user. It is.

A moving operation will be described as a specific example of the operation processing of the display object. FIG. 7 is an explanatory diagram illustrating a moving operation of the display object.
When the user touches two places at the same time on the touch pad unit 111 of the operation pad 100, the mode for moving to the display object is set. Further, when the user slides one of the touches, the display object 201 is moved and displayed in the sliding direction on the screen of the display unit 21.

  In order to indicate which display object the operation pad 100 corresponds to, when the operation pad 100 is touched, the corresponding display object blinks or the color is darkened, so that the user is discriminated. You may be able to do it. Further, thumbnails of corresponding display objects may be displayed at the corners of the operation pad 100.

  Next, in step S7, when the operation target is not a display target (window or object), the operation target is not specified, so the operation target is a cursor. Accordingly, an operation is performed on the cursor based on the touch operation performed on the operation pad 100 (step S9). That is, the operation determination unit 52 determines the operation content of the cursor movement based on the detection result of the operation detection unit 51, and sends the determination result to the operation execution unit 53. The operation execution unit 53 moves the cursor based on the content of the operation, and generates the image data. The display control unit 30 displays a cursor on the display unit 21 based on the image data.

  Regarding the cursor movement, the touch operation of the operation pad 100 and the degree of movement or scaling of the display object on the screen of the display unit 21 are set in advance, and can be arbitrarily set by the user. .

  In this way, when the operation pad 100 corresponding to the display object (window or object) is displayed one-to-one and each of the display objects is operated, a touch operation can be performed with the operation pad 100. Even if there are a plurality of users on the screen display, the operation can be performed smoothly without confusion. Further, the operation pad 100 itself can be operated with the operation pad 100, and the operation pad 100 can be smoothly operated without any confusion even if there are a plurality of users at a position desired by the user (e.g., an easy-to-operate position). And operability is improved. When the operation pad 100 performs an operation of designating a display object as an operation object (such as a long press operation of the display object), a one-to-one correspondence is displayed, so that the user can display the operation pad. Appearance operation is easy to understand.

[Second Embodiment]
In the first embodiment, operations on display objects such as movement, rotation, and enlargement / reduction have been described. However, in the second embodiment, a new image is additionally displayed.
A pen soft remote control button 101 is displayed on the operation pad 100, and touching this button enables character input.

FIG. 8 is a flowchart showing an operation process for handwritten input in the second embodiment.
This is a processing operation for the operation processing of the operation pad 100 in step S6 of FIG.
The operation determination unit 52 confirms whether the user has touched the pen touch remote control button 101 from the detection result of the operation detection unit 51 (step S11). If the pen touch remote control button 101 is touched, the operation execution unit 53 reads the character recognition application from the storage unit 40 and enters the handwritten character input mode (step S12).

  The user performs handwritten character input on the touchpad unit 111 of the operation pad 100 (step S13). The operation execution unit 53 performs character recognition by the application (step S14). And the recognized character is produced | generated as image data, and the display control part 30 displays the character image data on the display part 21 of the display apparatus 20 (step S15).

  As the character display position, for example, a character display clothing position may be designated from the operation pad 100 for cursor operation, and characters may be input from the operation pad 100 for cursor operation. In the case of (window or object), characters may be input from the operation pad 100 for operating the display object.

FIG. 9 is an explanatory diagram showing input of handwritten characters using the operation pad.
When the user touches the pen touch remote control button 101 on the operation pad 100, the pen touch remote control button 101 indicates that the display is reversed in black and white and the handwriting character input mode is entered. When the user inputs a character “W” by handwriting on the touch pad unit 111 of the operation pad 100, the character is recognized and “W” is displayed in the designated object 202 on the screen of the display unit 21. The

  In this way, handwritten character input using a finger or a touch pen is also possible, and handwriting input is performed after selecting a search area of the browser, whereby the handwritten input is recognized and search on the WWW is also possible. Furthermore, by directly inputting characters on the application, it is possible to add a meeting history to the display data on which the presentation was performed.

  If the size of the operation pad 100 is smaller than the area where characters are displayed, once the handwritten characters are input, the character size and character display position are changed to match the area where characters are displayed. By doing so, appropriate character arrangement becomes possible.

  The program that operates on the operation display device according to the present invention may be a program that controls a CPU (Central Processing Unit) or the like (a program that causes a computer to function) so as to realize the functions of the above-described embodiments related to the present invention. . Information handled by these devices is temporarily stored in RAM (Random Access Memory) during the processing, and then stored in various ROMs such as Flash ROM (Read Only Memory) and HDD (Hard Disk Drive). Reading, correction, and writing are performed by the CPU as necessary. In addition, a program for realizing the function of each component may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to perform processing of each unit. Good. Here, the “computer system” includes an OS and hardware such as peripheral devices.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to these embodiments, and the invention may be changed in design without departing from the gist of the present invention. Is also included.

In the present embodiment, the case where the present invention is applied to a flat table display has been described. However, the present invention can also be applied to a case where a plurality of users use a vertical large screen display. In this case, the operation pad may be set in advance so as to be displayed at a position touched by the user.
Of course, the present invention can also be applied to a vertically placed display. Further, the present invention can also be applied when a plurality of users use the tablet terminal.

10: Table display 20: Display device 21: Display unit 22: Touch input unit 30: Display control unit 40: Storage unit 50: Control unit 51: Operation detection unit 52: Operation determination unit 53: Operation execution unit 54: Character recognition unit 100: Operation pad

Claims (8)

An operation display device that displays on the display unit an operation pad that enables an operation by touching a display object displayed on the display unit,
An operation detection unit for detecting a touch operation;
An operation determination unit that determines an operation target and operation content in a touch operation based on a detection result of the operation detection unit;
When it is determined by the operation determination unit that a pad display operation for displaying the operation pad of the display object is performed, image data of the operation pad with the display object as an operation object is generated, and the operation An operation execution unit for generating image data for performing an operation on the display object by a touch operation on a pad;
A display control unit for displaying on the display unit based on the image data;
An operation display device comprising:   The operation display device according to claim 1, wherein the display object is an application window or an image object.   The operation display device according to claim 1, wherein the operation execution unit displays the operation pad near the periphery of the screen of the display unit closest to the display object.   The operation execution unit is provided with a button for operating itself on the operation pad, and enables a touch operation on the operation pad itself by a button operation on the operation pad. The operation display device according to any one of the above.   The operation determination unit determines that the operation pad is intended for an operation of moving a cursor displayed on the display unit when the pad display operation is performed on a portion other than the display object. The operation display device according to any one of claims 1 to 4. A character recognition unit for recognizing a character manually input on the operation pad when the operation content is determined to be in the character input mode by the operation determination unit;
The operation display device according to claim 1, wherein the operation execution unit generates a character recognized by the character recognition unit as a character image. An operation display method for displaying, on the display unit, an operation pad that enables operation by touching a display object displayed on the display unit,
An operation detection step for detecting a touch operation;
An operation determination step for determining an operation target and operation content in the touch operation based on a detection result of the operation detection step;
When it is determined in the operation determination step that a pad display operation for displaying the operation pad of the display object has been performed, image data of the operation pad with the display object as an operation object is generated, and the operation An operation execution step of generating image data for performing an operation on the display object by a touch operation on a pad;
A display control step for displaying on the display unit based on the image data;
An operation display method comprising:   The program for making a computer perform each step of the operation display method of Claim 6.

Images