JP5666546B2 - Information processing apparatus and image display program - Google Patents

Description

  The embodiment described in this specification relates to operation of an information processing apparatus having a touch panel display.

  Conventionally, when executing commands such as printing, scanning, and transmission for certain data on a touch panel display, it is common to press a hard button prepared in advance or a button provided at a predetermined position in the display screen Is.

  Further, in an operation using a touch panel display, when a finger is spread and placed on the screen, a frequently used icon appears near the finger, and a command is executed by selecting the icon.

  In the former general means, there are many procedures, not directly, such as selecting a target button after selecting target data. If the command appears in the vicinity of the latter finger, it is necessary to release the finger once from the screen, move it a little, and tap the screen again, resulting in an unnatural hand movement.

  The embodiment has been made to solve the above-described problems, and an object thereof is to provide a user interface capable of selecting commands and data with a more natural operation.

  The information processing apparatus according to the embodiment includes a touch panel display and a control unit. The control unit displays the touch panel display so that an image is displayed at a position corresponding to the fingertip of the hand based on the touch position of the touch panel display when the part touching the touch panel display is a pre-defined hand part. Control.

It is a figure which shows the external appearance of the desktop information processing apparatus of embodiment. It is a figure which shows an example of the hardware structural example of the desktop information processing apparatus of embodiment. It is a figure when the desktop type information processing apparatus of the embodiment is viewed from above, and a diagram showing a display example of a touch panel display. It is a figure which shows the user's operation example when displaying the icon for command execution. It is a figure which shows the example of a display of the icon for command execution, and the example of a display at the time of command selection. It is a flowchart which shows the operation example of the desktop information processing apparatus of embodiment. It is a figure which shows the structural example at the time of installing a several sensor unit so that it may align in each edge of a desktop information processing apparatus.

  In this embodiment, a desktop information processing apparatus (computer) will be described. The desktop information processing apparatus of this embodiment has a touch panel display on a flat top plate surface.

  The desktop information processing apparatus of the embodiment displays an operation command under the fingertip (thumb, forefinger, middle finger, ring finger, etc.) by bringing the palm of the hand into contact with the desktop touch panel display. The commands to be displayed include, for example, printing, scanning, transmission (sending to the cloud, e-mail, FAX, etc.), keyboard display, handwriting line editing, etc., which can be set by the user.

  The distance and angle from the palm to the fingertip and whether to use the right hand or the left hand can be set for each user and stored in advance in a storage unit such as a hard disk drive. In addition, since the operation of placing the palm on the screen can be done in a natural flow, the user can perform blind operations such as “tap the palm and tap the index finger” without having to look at the screen each time if the user learns the correspondence between each finger and the command. Is also possible. These commands vary depending on the data displayed at the place where the hand is placed, and the commands that can be used at that time are displayed.

  Hereinafter, aspects of the present embodiment will be described with reference to the drawings. FIG. 1 shows an external appearance of a desktop information processing apparatus 100 according to the embodiment, and FIG. 2 is a block diagram showing an internal hardware configuration of the desktop information processing apparatus 100. As shown in FIG. 1, the table-type information processing apparatus 100 is a table-type (table-type) information processing apparatus, and a large touch panel display 50 for operation display is arranged on the top plate surface. Using this touch panel display 50, a plurality of users can use it simultaneously. When a plurality of users use it, an area that can be used is divided for each user, and the user uses each area. In addition, discussions, presentations, business negotiations, and the like can be performed by a plurality of people using the desktop information processing apparatus 100.

  As shown in FIG. 2, the desktop information processing apparatus 100 includes a processor 10, a DRAM (Dynamic Random Access Memory) 20, a ROM (Read Only Memory) 30, an HDD (Hard Disk Drive) 40, a touch panel display 50, a network I / O. F (Interface) 60 and sensor unit 70 are provided. These devices exchange control signals and data with each other via a communication bus 80. The processor 10 is an arithmetic processing unit such as a CPU (Central Processing Unit), for example, and loads a program stored in the ROM 30 or the HDD 40 to the DRAM 20 and executes various processes according to the program. The DRAM 20 is a volatile main storage device. The ROM 30 is a non-volatile storage device that stores permanently, and stores a BIOS (Basic Input Output System) at the time of system startup. The HDD 40 is a nonvolatile auxiliary storage device that can be stored permanently, and data and programs used by the user are stored in the HDD 40.

  The touch panel display 50 includes an input unit of a capacitive touch panel and a display unit of a flat liquid crystal panel. The touch panel supports multi-touch that detects a plurality of simultaneous contacts, and can obtain coordinate values (X value, Y value) according to the contact position.

  The network I / F 60 is a unit responsible for communication with an external device, and includes a LAN (Local Area Network) board, a unit and mechanism for controlling short-range wireless communication, a FAX board connected to a public line network, and the like. The sensor unit 70 is a unit that detects an ID (identification) card possessed by a user and reads information described in the ID card. The read information is used for login authentication of the desktop information processing apparatus 100.

  FIG. 3A is a plan view when the desktop information processing apparatus 100 is viewed from above, and FIG. 3B is a diagram illustrating a display example of the touch panel display 50. The desktop information processing apparatus 100 allows multiple users to log in simultaneously, and can simultaneously display the desktop screens of multiple users. In this example, the sensor unit 70 is arranged at the center of each of the four side surfaces near the top plate (a total of four), and the user carrying the ID cards 200A to 200D approaches the sensor unit 70 in the proximity of the sensor unit 70. 70 reads the information in the ID card, and login authentication is performed. If the information in the ID card is registered in advance in the HDD 40 or an external authentication mechanism, the authentication conforms and the personal desktop screen is displayed in the direction the user is present (see FIG. 3B). ). The direction in which the user is present is determined by which sensor unit 70 the ID card information is acquired from.

  The touch panel display 50 also displays a boundary indicating a range that can be used by each individual. In the example of FIG. 3B, a broken line is used, but it is similar to this, and any mode may be used as long as the usage range can be understood by the user. In addition, the example of FIG. 3B is an example in which four users are using at the same time, the work areas of each user are areas R1 to R4, and the user who has the ID card 200A is The area R1 is used as a work area. Similarly, the user who owns the ID card 200B uses the area R2, the user who owns the ID card 200C uses the area R3, and the user who owns the ID card 200D uses the area R4. To do.

  As shown in FIG. 3B, the touch panel display 50 displays a desktop screen so as to be in a normal orientation for the user in accordance with the position of the user. That is, the desktop screen is displayed in such a direction that the user is at the bottom of the screen and the opposite side is at the top of the screen. Further, the desktop screen at the time of login may be displayed in the state when the user logged out last time, or may be displayed in the default state defined for each user in advance.

  The command selection method of this embodiment will be described with reference to FIGS. When the touch panel display 50 recognizes that the hand has been placed on the screen, the touch panel display 50 displays an icon (image) for each command for operating the data displayed there at the position of each finger. (Refer to the upper diagram of FIG. 5). Here, the orientation of the user with respect to the screen is detected and displayed in an appropriate direction with respect to the palm. The display position of the icon is adjusted in advance according to the palm size and dominant hand of each user.

  For example, for each piece of information (data type information) for distinguishing image data, text data, data of a specific application, etc., commands corresponding to the thumb to the ring finger (may be from the thumb to the little finger) are stored in the HDD 40. ing. Data regarding each finger is indicated by a distance from a base point (described later) of the wrist and an angle with reference to the X-axis direction. These data are stored for each user. Thereby, the processor 10 can change the kind of command according to the displayed data. When a plurality of different types of data are displayed, the processor 10 changes the display command according to the data at the contact position of the carpal part. The order (assignment to each finger) can be changed according to the user's preference.

  FIG. 5 is a display example in the case where there is a contact of the carpal portion in the area where the image data is displayed. When the touch panel display 50 detects the contact of the wrist (see FIG. 4), the touch panel display 50 displays a command icon for image data at an appropriate position of each fingertip according to the shape of the user's hand according to the control of the processor 10. Is displayed (see the upper part of FIG. 5). In this example, icons for character input, printing, transmission to an external device or cloud, and deletion are displayed at the position of each fingertip (the mode is not limited to this). In this state, when the index finger is tapped, a print command is issued and the image data is printed (see the lower part of FIG. 5). In addition, when a touch of the carpal part is detected in a place where there is no data (immediately on the desktop) or on undefined data, a default command execution icon is displayed at the fingertip position. As this default command, for example, a keyboard operation at the thumb position, a list of applications that can be used at the index finger position, data selection at the middle finger position, logout at the ring finger position, and the like can be considered.

  By adopting such a configuration, it is possible to input a gesture of a command based on palm detection. Moreover, the implementation which maintains the state which displayed the icon for a fixed period (for example, 5 second) may be sufficient even if it releases | separates a wrist part. In this case, if an icon is selected within this period, the corresponding command is executed.

  FIG. 6 is a flowchart illustrating an operation example of the desktop information processing apparatus 100.

  The center unit 70 detects an ID card owned by the user in the vicinity and reads the information. The processor 10 performs login authentication by determining whether this information is information registered in advance in an external authentication mechanism or the HDD 40 (ACT001). When the authentication is conforming (ACT001, conforming), the processor 10 obtains information on where the login user is (ACT002). This information is derived depending on which center unit 70 has detected.

  The processor 10 acquires parameter information of the login user from the HDD 40 (ACT003). This parameter information includes information on the distance and angle of each finger from the base point of the wrist. The parameter information is expanded in the DRAM 20 that can be accessed at higher speed. The parameter information also includes layout data for the user's desktop screen, image data such as icons, and the touch panel display 50 is in a proper orientation according to the position of the logged-in user according to the information. Display the desktop screen.

When the touch panel display 50 detects any contact (ACT004, Yes), the processor 10 determines whether or not the contact is the wrist part of the hand (ACT005). This determination is, for example, any one or a combination of the following determinations.
(1-1) When the hand portion touches a flat surface, the processor 10 makes contact with a shape having a certain area, and therefore the processor 10 determines whether or not there are a plurality of simultaneous contacts within a certain range for a certain period. In the present embodiment, the period is set to about 1 second, the range is set to an area or shape range about the human wrist, and the number of simultaneous contacts is set to 5 to 6 or more, but the mode is not limited to this.
(1-2) For each user, the shape (image and feature point) of the carpal portion is stored in the HDD 40 in advance, and when there is a contact, the processor 10 determines the contact shape and the shape in the HDD 40. Judgment is made by comparison using a pattern matching technique or the like. In this case, the panel of the touch panel display 50 is provided with an image sensor for reading together with a light emitting element for display, and also has a reading function (scanner function). Since the shape of the wrist portion is stored for each user, when a plurality of users are using the desktop information processing apparatus 100 at the same time, the processor 10 detects which user has touched the wrist portion. be able to.

If it is determined in ACT005 that the contact site is not the carpal part (ACT005, No), other processing is performed (ACT011), and the process proceeds to ACT010. When the contact part is the carpal part (ACT005, Yes), the processor 10 determines the direction of the carpal part (ACT006). If you do not specify the direction of the fingertip of the hand when detecting contact with the wrist, the icon may be displayed in the direction opposite to the original fingertip direction. Is determined. This determination is, for example, any one or a combination of the following determinations.
(2-1) The direction is determined according to the usable range of the user. Describing with reference to FIG. 3B, for example, when the contact of the wrist portion is detected in the region R3, it is assumed that the fingertip is in the upward direction on the paper surface of FIG. deal with. In the case of R1, it is assumed that the fingertip is in the downward direction on the paper surface of FIG. 3B from the wrist contact position, R2 is in the right direction, and R4 is in the left direction. At this time, the processor 10 considers that the user using the area has touched the wrist.
(2-2) For the first time only, touch with the fingertip after the carpal contact. As a result, the direction of the carpal portion is specified, and thereafter, the direction is determined even when there is no fingertip touch.
(2-3) When the touch panel display 50 has a reading function (scanner function), the direction of the wrist portion is determined by the shape of the wrist portion. Since the contact portion has a shape close to an ellipse that is slightly depressed in the finger direction, the processor 10 can determine the finger direction by this shape recognition.

  In addition to the above, by using the human body communication technology, information in the ID card can be read through the human body through contact with the wrist. In this case, it is necessary to mount a human body communication module. Thus, the processor 10 can specify which user is the user who has contacted the wrist, and can specify the direction according to the position of the user (the position detected by the sensor unit 70). it can. The ID card may be stored in a pocket or may be lowered from the neck. Even in these states, communication is possible through the human body.

  Next, the processor 10 displays a predefined icon at the position of each fingertip based on the angle and distance of the fingertip acquired by ACT003 (ACT007). At this time, the processor 10 specifies the type of data displayed at the position where the hand touched, and acquires an icon corresponding to the data type from the HDD 40 and causes the touch panel display 50 to display the icon.

  When the shape of the carpal contact is specified as in (1-2) above, an icon is drawn at the position of the angle and distance with the center of gravity of the shape as the base point. When multiple points are detected within a certain range as in (1-1) above, the most central detection point is very good. In the case of (1-1), the average value of the X and Y coordinate values of each detection point may be calculated, and the calculated coordinate value may be used as the base point. In the case of (1-1), the center coordinates of the two most distant points may be used as the base point. These mounting examples are merely examples, and various methods can be considered for specifying the base point. In this example, the icon drawing position is determined based on the angle and distance from the base point, but the aspect is not limited to this. Relative coordinate values from the position (coordinates) of the base point are stored in the HDD 40 for each fingertip, and when contact with the wrist is detected, the coordinate value serving as the base point and the coordinate value stored in the HDD 40 are used. The implementation may be such that an absolute coordinate value is calculated and an icon is drawn at the coordinate.

  The icon display naturally takes into account the direction specified in ACT006, and the icon image and the display position are rotated and displayed at 0 °, 90 °, 180 °, and 270 ° with the base point as the center point. As shown in FIG. 7, a more precise angle can be detected by arranging the sensor units 70 so as to be aligned along the sides of the desktop information processing apparatus 100. In this case, the detection range of each sensor unit 70 is narrowed. The processor 10 calculates an angle based on the position of the sensor that has detected the ID card 200 and the contact position of the wrist, and controls the touch panel display 50 so that an icon image is displayed at the position and angle corresponding to the angle. . Further, when the contact position of the wrist part is far from the detection sensor, the elbow part of the arm is not bent so much, and when close, the elbow part of the arm is bent. For this reason, the implementation may be such that the angle is calculated according to the distance between the sensor detection position and the wrist part. In addition, when the shape of the carpal contact is specified as in (1-2) above, the angle can be detected more precisely, and thus the rotation is displayed according to the detected angle.

  The touch panel display 50 detects the presence / absence of contact on the displayed icon (ACT008). When there is contact, that is, when there is selection and pressing with any finger (ACT008, Yes), the processor 10 executes a command associated in advance with the icon with contact (ACT009). The processes of ACT004 to ACT009 and ACT011 are performed until the user logs out (ACT010, No loop).

  In the above-described embodiment, an example in which the contact of the wrist (wrist) is detected and an icon is displayed at the fingertip position based on this position has been described. Alternatively, a method may be used in which a portion (small finger ball portion) on the outer surface applied to the touch panel display 50 is brought into contact with the contact position as a reference. In this case, an icon is displayed at each fingertip position when the upper thumb tip is brought into contact with the touch panel display 50 and the hand is placed in a lying posture. Alternatively, one or both of the tip of the thumb and the tip of the little finger may be brought into contact, and icons may be displayed at other fingertip positions based on the contact position. There are various modes other than these. That is, in this embodiment, any part of the hand that is defined in advance can be used as a reference.

  In the present embodiment, the command is executed and the operation is described. However, the present invention is not limited to the command execution. For example, a preview image obtained by reducing an image such as a photo at the fingertip position may be displayed, and an image selected by pressing the fingertip may be enlarged and displayed in an actual size. In order to reproduce audio data or a movie, etc. It can also be used for selection. These are data selections, not command selections.

  In the present embodiment, the description has been made on the assumption that an icon (image) is displayed at the fingertip position. However, an implementation in which text is displayed and selected may be used. When text data is displayed, it is copied as image data, so that the text data can also be regarded as an image.

  In addition, for example, in the state shown in the upper diagram of FIG. 5, it is possible to mount the finger and the icon so that the correspondence relationship between the finger and the icon is shifted by touching the touch panel display 50 and sliding the finger as it is. For example, when the index finger is slid leftward in the state shown in the upper part of FIG. 5, the print icon is shifted to the thumb position, the transmission icon is shifted to the index finger position, and the delete icon is shifted to the middle finger position. At the position of the ring finger, other command execution icons that could not be displayed are displayed. Similarly, when the index finger is slid in the right direction, the shift is in the right direction. In the case of such a shiftable configuration, the correspondence relationship between the data relating to the fingertip and the data relating to the command stored in the HDD 40 is the default, and each data used during execution of processing developed in the DRAM 20 The correspondence relationship is variable.

  In addition, the display of the fingertip position may be configured to have a hierarchy. For example, an icon indicating a folder (a mode is not limited to this) is displayed at each fingertip position at the time of detecting a wrist, and when a fingertip press is detected, an icon indicating a command or data in the folder or a subfolder is displayed. It is possible to implement the display at the position of each fingertip.

  An image display program or a command execution program for causing the computer to execute the above operation may be provided. This embodiment will be described as an example. The image display program and the command execution program are stored in advance in the HDD 40, and the processor 10 reads out the program, expands it in the DRAM 20, and executes the calculation.

  In the present embodiment, the function for implementing the invention is recorded in advance in the apparatus. However, the present invention is not limited to this, and the same function may be downloaded from the network to the apparatus. You may install what was memorize | stored in the apparatus. The recording medium may be any form as long as the recording medium can store the program and can be read by the apparatus, such as a CD-ROM. In addition, the function obtained by installing or downloading in advance may be realized in cooperation with an OS (operating system) inside the apparatus.

  The control unit corresponds to a configuration including at least the processor 10, the DRAM 20, and the communication bus 80 of the embodiment. A program that operates in cooperation with hardware such as the processor 10, the DRAM 20, and the communication bus 80 is stored in advance in the HDD 40 (or may be the ROM 30), and is executed by the processor 10.

  As described in detail above, according to the aspect of the present embodiment, an image (for example, a command execution icon) can be displayed by a natural operation of the user. As a result, the command and data can be selected and pressed with a more natural movement.

  The present invention can be implemented in various other forms without departing from the spirit or main features thereof. Therefore, the above-described embodiment is merely an example in all respects and should not be interpreted in a limited manner. The scope of the present invention is indicated by the scope of claims, and is not restricted by the text of the specification. Further, all modifications, various improvements, alternatives and modifications belonging to the equivalent scope of the claims are all within the scope of the present invention.

  10 processor, 20 DRAM, 30 ROM, 40 HDD, 50 touch panel display, 60 network I / F, 70 sensor unit, 80 communication bus, desktop information processing apparatus 100 (information processing apparatus).

Claims (8)

A touch panel display;
Recognizing touch on the touch panel display,
If the part touching the touch panel display is a predefined hand part, specify the direction of the part of the hand that touched,
Based on the orientation of the identified hand part, identify the position of the fingertip of the hand,
A control unit that controls the touch panel display such that the image is displayed at the position of the fingertip which is the specific,
An information processing apparatus. The information processing apparatus according to claim 1,
The controller controls the orientation of the identified hand part and the touch panel display. Based on the touch position, the distance from the base point of the hand region, which is predefined for each user, and Using the angle data, specify the fingertip position of the hand
Information processing device. The information processing apparatus according to claim 1 or 2,
The controller controls one or a plurality of areas used by a user in the touch panel display. Based on the contact area created in the area, specify the orientation of the part of the touched hand Do
Information processing device. The information processing apparatus according to claim 1 or 2,
The control unit determines a direction in which the contact is detected after the hand part is in contact with the direction of the hand part. Kiss
Information processing device. The information processing apparatus according to any one of claims 1 to 4 ,
The information processing apparatus is a table-type information processing apparatus, and the touch panel display is arranged so that the top surface of the top panel is a display surface.
The said control part controls the said touch panel display so that the image according to each user is displayed in the position according to each user's fingertip, when the simultaneous login of multiple users is performed . The information processing apparatus according to any one of claims 1 to 5 ,
The control unit controls the touch panel display so that an image according to a type of data displayed at a position where the part is in contact is displayed. The information processing apparatus according to any one of claims 1 to 6 ,
The hand part is a carpal part. A program for causing a computer having a touch panel display to execute,
Recognizing touch on the touch panel display,
Site in contact with the touch panel display, if a portion of the hand previously defined things, identify the orientation of portions of the hand in contact,
Based on the orientation of the identified hand part, identify the position of the fingertip of the hand,
The image display program for a process for controlling the touch panel display such that the image is displayed at the position of the fingertip which is the specified, causes the computer to execute.