JP5947999B2 - Method, electronic device and computer program for improving operation accuracy for touch screen - Google Patents

Description

  The present invention relates to a technique for improving the operation accuracy for a touch screen, and more particularly to a technique for improving the operation accuracy of a small touch screen.

  The touch screen mounted on the electronic device employs a laminated structure of a display and a transparent touch panel, and direct input instructions can be given to objects displayed on the display with a finger or electronic pen, enabling intuitive operation by the user To. The finger has a larger area (contact area) in contact with the touch surface than the electronic pen. In a capacitive touch panel, the coordinates are recognized by calculating the coordinates of the geometric center and the center of gravity of the contact area of the finger in contact with the display surface.

  Since the user understands the detection principle of the touch screen, the user performs a touch operation so that the center of the contact area of the finger is positioned on the target object. However, the contact area has an area spread according to the size of the finger and the contact pressure, and exists on the back side of the finger, so that it becomes difficult for the user to perform an accurate operation as the target object is smaller. Non-Patent Document 1 has published research results on structural error offsets that occur in current touch screens. This document describes the following.

  In the capacitive touch sensor, when the user designates an object, the center part of the fingernail is used as a reference, so the coordinates detected by the device from the contact area and the user intends to position the finger. An offset occurs between the coordinates. Since the contact area changes depending on the angle and pressure of the finger with respect to the touch screen, the size of the target needs to be increased in order for the touch screen to accurately detect the object intended by the user. Therefore, it has been reported that a device that detects coordinates from a finger image captured using a camera can detect a position intended by a user more accurately than a device that detects a target at the center of a finger contact area. .

  Patent Document 1 discloses a touch panel in which two fisheye lenses / cameras are mounted on a bezel of a display device of a computer and a finger pointing position is detected based on the principle of triangulation. Patent Document 2 discloses a touch panel device that reduces touch errors by photographing fingers with four cameras arranged around a touch screen. The area of the finger is estimated from an image obtained by photographing the finger approaching the touch screen by the camera. If the finger area is larger than a predetermined value, the GUI component is enlarged and displayed. In either case, when the finger is not approaching, the GUI is not displayed on the touch screen.

JP 2010-282463 A JP 2011-134111 A

Christian Holz and Patrick Baudisch, "Understanding Touch", [online], 2011, [searched January 18, 2014], Internet <URL: http://www.christianholz.net/2011-chi11-holz-baudisch -understanding_touch.pdf>

  As Non-Patent Document 1 points out, even if an input device using a camera can be operated with higher accuracy than a conventional capacitive touch screen, the finger is placed directly above the portable electronic device. It is difficult to mount a camera for taking pictures. In the case of detecting coordinates only by an image as in Patent Document 1, it is necessary to mount two cameras. In the method of Patent Document 2, since icons are not displayed before the finger approaches, if the number of icons increases, it is expected that the target icon will not be displayed when the finger is approached, and icons are always displayed on the screen. If you want to, you can not employ.

  A tablet terminal and a smartphone are typically equipped with one camera that can switch and shoot a user's face and landscape. Some recent notebook personal computers (notebook PCs) are equipped with a single camera and a touch screen that can photograph a user's face. In such portable electronic devices, it is required to improve the operation accuracy of the touch screen while suppressing an increase in cost.

  Therefore, an object of the present invention is to provide a method for improving the operation accuracy of a touch screen mounted on an electronic device. Another object of the present invention is to provide a method for reducing the size of a touch screen object mounted on an electronic device. Furthermore, the objective of this invention is providing the electronic device and computer program which employ | adopted such a method.

  In one aspect of the present invention, the electronic device includes a touch screen that detects coordinates from a contact area of a finger with respect to the touch surface. The electronic device generates detected coordinates in response to a touch operation on the touch screen. The electronic device takes a picture of a finger touching with the camera. Subsequently, the electronic device calculates a correction amount based on the posture of the finger specified from the photographed image of the finger, and outputs target coordinates calculated from the detected coordinates and the correction amount.

  As a result, it is possible to improve the operation accuracy by correcting the offset generated due to the posture of the finger performing the touch operation. In addition, the object to be touched can be reduced. The electronic device can be either a smartphone, a tablet terminal, or a notebook PC. The touch screen can be either a capacitance type, a FTIR (Frustrated Total Internal Reflection) type, or a resistance type. If the camera is a wide-angle lens camera or fish-eye lens camera, it can be used for both face and finger photography, and even if the edge frame of the touch screen is on the same plane as the touch surface, the finger to be touched can be photographed. Can do.

  In the calculation of the correction amount, the target coordinates can be set directly below the line-of-sight target assumed at a predetermined position of the finger. At this time, the line-of-sight target can be set on the center line of the width of the finger formed by the projection of the finger on the touch surface and closer to the fingertip side than the bottom of the finger. Furthermore, in the calculation of the correction amount, it is possible to assume the detection coordinates immediately below the characteristic part of the finger included in the captured image. At this time, the characteristic part of the finger can be provided at the bottom of the nail. In the calculation of the correction amount, the pitch angle formed by the finger axis and the touch surface specified from the captured image can be calculated. At this time, the distance between the camera and the detected coordinate can be calculated to calculate the length of the nail specified from the captured image. One camera can be obtained by calculating the distance between the camera and the detected coordinates.

  Further, in the calculation of the correction amount, the roll angle around the finger axis specified from the photographed image can be calculated. At this time, the distance between the camera and the detected coordinates can be calculated, and the height of the finger relative to the touch surface specified from the captured image can be calculated. In the above configuration, the pitch direction inclination and the roll direction rotation may occur simultaneously. The present invention is particularly effective when applied to a touch screen mounted on a small electronic device such as a smartphone or a tablet terminal.

Another aspect of the present invention is a function in which an offset corresponding to a difference between a target coordinate that a user intends to input to an electronic device and a detected coordinate detected by a touch screen is OFS, and a variable corresponding to a finger posture is u. Formula (a) is prepared, where f (u) is the coefficient and r is the coefficient.
OFS = r × f (u) (a)

  Subsequently, the touch screen generates detected coordinates, and the finger that is touching with the camera is photographed. Subsequently, the offset value is calculated by substituting the value of the variable u calculated based on the posture of the finger specified from the photographed image of the finger into the equation (a), and the target coordinate calculated from the detected coordinate and the offset value is output. As the coefficient r, a value that can be averagely applied to a plurality of subjects may be registered in advance, or a value that can be applied to individual users may be registered.

When the variable u is the pitch angle β, the offset in the pitch direction is Δy, and the coefficient r is the coefficient p, the equation (a) is
Δy = p × cos β (b)
It can be. When the variable u is the roll angle γ, the offset in the pitch direction is Δx, and the coefficient r is the coefficient q, the equation (a) is
Δx = q × tan γ (c)
It can be.

  The coefficient r displays the target object on the touch screen, calculates an offset value from the difference between the detected coordinates detected in response to the touch operation on the target object and the coordinates of the target object, and calculates the value of the variable u from the captured image. Can be calculated by substituting the offset value and the value of the variable u into equation (a). Therefore, even if the finger shape and the position of the line-of-sight target are different for each user, it is possible to perform highly accurate correction by calculating the coefficient r for each user.

  According to the present invention, a method for improving the operation accuracy of a touch screen mounted on an electronic device can be provided. Furthermore, according to the present invention, it is possible to provide a method for reducing an object of a touch screen mounted on an electronic device. It is a further object of the present invention to provide an electronic device and a computer program that employ such a method.

It is a figure explaining a mode when touch-operating the touch screens 11 and 21 of the smart phone 10 and the tablet terminal 20 with a finger. It is a figure explaining a mode that offset occurs in touch operation by a finger. It is a figure for demonstrating the method of correct | amending the coordinate which the touch screen 11 detected. It is a figure for demonstrating the method of correct | amending the coordinate which the touch screen 11 detected. It is a figure explaining the method to produce | generate the target coordinate on a XY rectangular coordinate according to the yaw angle (alpha) from offset (DELTA) x, (DELTA) y on a xy rectangular coordinate. FIG. 3 is a diagram for explaining a hardware configuration of a smartphone 10. 2 is a functional block diagram of a coordinate correction system 200. FIG. 5 is a flowchart for explaining the operation of the coordinate correction system 200. 10 is a flowchart for explaining another operation of the coordinate correction system 200. 6 is a diagram for explaining another operation of the coordinate correction system 200. FIG.

  FIG. 1 is a diagram for explaining a state when the touch screens 11 and 21 of the smartphone 10 and the tablet terminal 20 that are examples of the portable electronic device according to the present embodiment are operated with a finger. The smartphone 10 and the tablet terminal 20 have camera modules 13 and 23 mounted on edge frames 15 and 25 surrounding the touch screens 11 and 21. The smartphone 10 illustrates a single touch operation with a single finger, and the tablet terminal 20 illustrates a multi-touch operation with a plurality of fingers. The surfaces of the touch screens 11 and 21 that perform the touch operation are referred to as touch surfaces 11a and 21a.

  FIG. 2 is a diagram for explaining how a structural offset occurs in the touch operation on the touch surface 11a. The user touches the rectangular icon 111 displayed on the touch screen 11 with the finger 150. Since the user knows that the touch screen 11 detects the coordinates at the center of the contact area 109 where the finger 150 and the touch surface contact, the touch operation is performed so that the center of the contact area 109 is positioned at the center of the icon 111. To do. Here, the central portion of the icon 111 is a target coordinate when the user positions the finger, and this is referred to as a target coordinate 105.

  However, it is known from the experience of daily touch operations that the icon 111 may not be selected when the area of the icon 111 is small. In the research of Non-Patent Document 1, it is pointed out that the user implicitly positions the center of the surface of the nail 151 at the target coordinate 105 at this time. A predetermined position on the surface of the nail 151 or the skin that is assumed to be a marker without being conscious when the user positions the finger at the target coordinates 105 is referred to as a line-of-sight target 155. At this time, it is assumed that the target coordinates 105 are present on a perpendicular line from the line-of-sight target 155 to the touch surface.

  However, the present invention can correct not only when it is assumed that the line-of-sight target 155 is present at the center of the nail 151 but also when there is an individual difference in the position of the line-of-sight target 155. Here, the coordinates detected by the touch screen 11 are referred to as detected coordinates 107. Since the detection coordinate 107 is the geometric gravity center or center of the contact area 109 as a principle of the touch screen 11, an offset OFS is generated between the detection coordinate 107 and the target coordinate 105 due to the shape and posture of the finger. To do.

  When the offset OFS is larger than the size of the icon 111, the touch operation on the icon 111 fails or is erroneously performed on an adjacent object. Although the line-of-sight target 155 is hidden in the user's consciousness, the cause of the target coordinate 105 occurring at a position shifted from the detected coordinate 107 is created, and the offset OFS is calculated by assuming the position. The principle of the present embodiment is configured in FIG.

  3 and 4 are diagrams for explaining a method of correcting coordinates detected by the touch screen 11 using image data (image data) taken by the camera module 13. In FIG. 3, XY orthogonal coordinates are defined along the short side and the long side of the rectangular touch screen 11. The touch screen 11 outputs the detected coordinates 107 calculated from the contact area 109 of the finger 150 to the system in XY orthogonal coordinates. The detected coordinate 107 exists at a distance d from the camera module 13. As will be described later, the distance d is used to calculate the nail length s, the nail width n, the finger height h1, and the like from finger image data captured by one camera module 13.

  In FIG. 3, the y-axis is set in the length direction of the finger 150 to define the xy orthogonal coordinates of the finger. An angle formed by the y axis of the finger and the Y axis of the touch screen 11 is referred to as a yaw angle α. FIG. 4 shows the range of the finger 150 related to the touch operation from the fingertip to the vicinity of the first joint. In FIG. 4, when the cross section of the finger is regarded as an ellipse, a line passing through the center of the finger in the length direction (y-axis direction) of the finger 150 is referred to as a finger axis 160.

  4A and 4C are plan views of a state in which the finger is placed on the touch surface 11a so that the finger axis 160 is parallel to the touch surface 11a, as viewed from directly above. 4A shows the front of the nail 151, and FIG. 4C shows the finger 150 slightly rotated rightward (roll direction) about the finger axis 160. In FIG. 4A, in the projection onto the touch surface 11a, the length formed by the position where the line 154 passing through the bottom portion 157 of the nail and the surface on both sides of the skin intersects with the surface of the both sides of the skin is formed. Define as w1.

  In the projection onto the touch surface 11a, the length of the nail 151 in the direction perpendicular to the finger axis 160 is defined as a nail width n. The center of the finger width w1 at the bottom 157 of the nail is referred to as the finger width center 162, and the center of the nail width n is defined as the nail width center 168. In projection onto the touch surface 11a, the line passing through the finger width center 162 and the finger axis 160 are considered to coincide with each other, and the finger width center 162 and the nail width center 168 are assumed to coincide with each other at the finger bottom 157.

  In the projection on the touch surface 11a, the length on the finger axis 160 from the line 154 passing through the bottom 157 of the nail and parallel to the touch surface 11a to the tip of the skin of the finger 150 or the tip of the nail 151 is the length of the nail. S. As an example, in the projection onto the touch surface 11a, the surface of the nail 151 corresponding to the position where the finger axis 160 and the center line 152 of the nail length intersect is assumed to be the line-of-sight target 155. In the projection onto the touch surface 11a, the y-axis is defined parallel to the finger axis 160 through the finger width center 162, and the x-axis is defined parallel to the line 154 through the finger width center 162. The xy orthogonal coordinates are parallel to the touch surface 11a.

  In FIG. 4C, in the projection onto the touch surface 11a, the length formed by the position where the line 154 passing through the bottom portion 157 of the nail and the surfaces of both ends of the skin intersect with each other forms the width of the finger. It is defined as w2. Both the finger widths w1 and w2 correspond to the width formed by the line 154 passing through the bottom 157 of the nail in the projection of the skin onto the touch surface 11a. The finger width w1 from the finger shape corresponds to the maximum value of the finger width w2. The surface of the skin or the nail where the center of the finger width w2 in the line 154 passing through the bottom 157 of the nail is located is referred to as the finger width center 164.

  In the calculation of the correction amount in the present embodiment, the finger width w2 is applied when the finger 150 rotates in the roll direction regardless of the roll angle γ. Assume that a line of sight target 155 corresponds to the nail 151 or the surface of the skin corresponding to the position where the finger axis 160 and the center line 152 of the length of the nail intersect in the projection onto the touch surface 11a corresponding to FIG. In the projection onto the touch surface 11a, the y-axis is defined parallel to the finger axis 160 through the finger width center 164, and the x-axis is defined parallel to the line 154 through the finger width center 164. Also in this case, the xy orthogonal coordinates are parallel to the touch surface 11a.

  FIG. 4B is a side view of the finger 150 not rotating in the roll direction as viewed from the x-axis direction. A line 156 that passes through the center 162 of the finger width and extends along the surface of the nail 151 in parallel with the finger axis 160 is inclined in the pitch direction with respect to the touch surface 11a. An angle formed by the line 156 and the touch surface 11a is referred to as a pitch angle β. Since the line 156 and the finger axis 160 can be treated as parallel, the pitch angle β can also be defined by the angle between the touch surface 11 a and the finger axis 160. Further, the pitch angle β may be defined as an angle between the touch surface 11a and the skin on the back side of the finger 150 facing the touch surface 11a.

Here, in FIG. 4B, the line-of-sight target 155 is the nail 151 corresponding to the position where the finger axis 160 and the center line 152 of the nail length intersect in the projection onto the touch surface 11a, as in FIG. 4A. Assuming the surface of However, the line-of-sight target 155 may be assumed on the surface of the nail 151 on the tip side of the finger bottom 157 on the line 156. In addition, it is assumed that the detection coordinate 107 exists on a perpendicular line extending from the center 162 of the finger width to the touch surface 11a. At this time, an offset Δy in the y-axis direction is generated between the target coordinate 105 immediately below the line-of-sight target 155 and the assumed detection coordinate 107. The offset Δy can be calculated by the equation (1) from the pitch angle β and the claw length s.
Δy = (s / 2) × cos β (1)

  FIG. 4D is a cross-sectional view of the cross section when the finger 150 is cut along a plane perpendicular to the touch surface 11a including the x axis in FIG. At this time, the nail center 168 is rotated by the roll angle γ from the vertical direction with respect to the touch surface 11a in FIG. At this time, since the finger 150 is not completely circular in cross section, the detection coordinate 107 is shifted from the target coordinate 105 in the x-axis direction. As in FIG. 4B, the target coordinate 105 is set just below the line-of-sight target 155, and the detected coordinate 107 is assumed just below the center 168 of the nail width at the position rotated by the roll angle γ. However, the line-of-sight target 155 may be assumed to be on the surface or skin of the nail 151 passing through the center 164 of the finger width and on the tip side of the finger bottom 157 in the projection onto the touch surface 11a. At this time, an offset Δx in the x-axis direction is generated between the detected coordinate 107 and the target coordinate 105.

The offset Δx can be calculated from the roll angle γ and the height h1 of the finger with respect to the touch surface 11a in the roll state using Equation (2).
Δx = (h1 / 2) × tan γ (2)
The direction of the offset Δx coincides with the roll direction, but from the viewpoint of the nature of the movable range of the finger, the right finger is normally rotated to the right and the left hand is rotated to the left as viewed from the user.

  When the finger 150 is inclined in the pitch direction and is rotated in the roll direction, the nail 151 corresponding to the position where the finger axis 160 and the center line 152 of the nail length intersect in the projection onto the touch surface 11a. Alternatively, the line-of-sight target 155 is set on the surface of the skin, and the offsets Δy and Δx can be calculated from the pitch angle β and the roll angle γ extracted from the image, respectively. The offsets Δx and Δy are values on the xy orthogonal coordinates, but the touch screen 11 needs to output values on the XY orthogonal coordinates.

When the y-axis of the finger 150 is tilted by the yaw angle α with respect to the Y-axis of the touch screen 11, the offsets Δx and Δy are orthogonal to each other in the formulas (3) and (4) as shown in FIG. The detected coordinates (X ₀ , Y ₀ ) can be corrected by projecting on the target coordinates (X ₁ , Y ₁ ) on the coordinates.
X ₁ = X ₀ −Δx × cos α (3)
Y ₁ = Y ₀ + Δy × cos α (4)

  FIG. 6 is a diagram for explaining a hardware configuration of the smartphone 10. The smartphone 10 includes elements such as a CPU 51, a system memory 53, an I / O interface 55, an SSD 57, a touch screen 11, a wireless module 59, and a camera module 13. Since many pieces of hardware constituting the smartphone 10 are well known, they will be described within the scope necessary for understanding the present invention. The SSD 57 stores software that constitutes a main element of the coordinate correction system 200 (FIG. 7).

  The touch screen 11 has a structure in which a display such as a liquid crystal display or an organic EL and a transparent touch panel are stacked. When a touch operation is performed on an object displayed on the display with a finger, the touch panel detects the finger coordinate and outputs a detection coordinate 107. To do. The present invention can be applied to a touch screen that outputs the coordinates of the center or the center of gravity of the contact area 109 with which the finger 150 is in contact, such as a capacitance type, an FTIR type, or a resistance film type. A touch screen including this type of touch panel generates a structural offset between the target coordinates 105 and the detection coordinates 107 as described with reference to FIG.

  The camera module 13 includes an image sensor such as a CMOS or a CCD and an image signal processor. The camera module 13 is equipped with a fisheye lens or an ultra-wide-angle lens, and can photograph a hemispherical object having a horizontal direction of 360 degrees and a vertical direction of 180 degrees. The number of camera modules 13 in this embodiment may be one. Therefore, the camera module 13 can be shared with a camera module that is normally mounted on the smartphone 10 in order to photograph a user's face and landscape.

  The smartphone 10 employs a so-called full flat display system in which the edge frame 15 and the touch surface 11a are on the same plane. Many tablet devices also use the full flat display method. Since the camera module 13 is equipped with a fisheye lens or an ultra-wide-angle lens, even if it is mounted on the edge frame 15 of the full flat display, the face and finger of the user who performs the touch operation can be photographed simultaneously. If the camera module can be mounted on the edge frame so that the lens axis faces the finger that performs the touch operation, a standard lens camera or a wide-angle lens camera can also be employed.

  FIG. 7 is a functional block diagram for explaining the configuration of the coordinate correction system 200. The image data acquisition unit 201, the correction amount calculation unit 203, the registration unit 204, the coordinate acquisition unit 205, the application execution unit 207, and the image data output unit 209 include the hardware such as the CPU 51 and the system memory 53 illustrated in FIG. Is configured by the cooperation of the software stored. When the operation starts, the camera module 13 transfers the image data including the image of the finger 150 performing the touch operation and the face of the user to the image data acquisition unit 201 as an example at a frame rate of 30 fps. The touch screen 13 generates the detection coordinates 107 from the contact area 109 and outputs the detection coordinates 107 to the coordinate acquisition unit 205 while the finger 150 is in contact with the touch surface 11a at a predetermined sampling cycle.

  When receiving the detected coordinates 107 generated by the touch screen 11 from the coordinate acquisition unit 205, the image data acquisition unit 201 instructs the camera module 13 to start transferring image data. Further, when the detected coordinates 107 are not received for a predetermined time, the camera module 13 is instructed to end the transfer of image data. The image data acquisition unit 201 sends correction data obtained by combining the image data received from the camera module 13 and the detected coordinates 107 received from the coordinate acquisition unit 205 to the correction amount calculation unit 203.

  The correction amount calculation unit 203 has an object recognition function and, if necessary, a face recognition function. The correction amount calculation unit 203 calculates the distance d shown in FIG. 3 from the received correction data. When the correction amount calculation unit 203 calculates the distance d and performs a touch operation using the object recognition function, the yaw angle α, pitch angle β, roll angle γ, nail length s, nail width n of the finger 150 The finger widths w1 and w2 and the finger height h1 are calculated. The correction amount calculation unit 203 recognizes the user's face and the identifier of the user who is currently performing the touch operation and the identifier of the finger, the nail length s that does not change depending on the posture of the finger that is performing the touch operation, The width n is registered in the registration unit 204.

  The correction amount calculation unit 203 registers the nail length s and the nail width n in the registration unit 204 every time a new user performs a touch operation. The correction amount calculation unit 203 calculates offsets Δx and Δy using equations (1) to (4), further calculates the target coordinates 105, and the coordinate acquisition unit 205 sends the detected coordinates 107 to the image data acquisition unit 201. The time stamp added at the time is sent to the coordinate acquisition unit 205.

  In another example, the correction amount calculation unit 203 holds formulas (5) and (6) which will be described later, and receives data from the application execution unit 207 to register data necessary for correction. The coordinate acquisition unit 205 adds a time stamp and sends it to the image data acquisition unit 201 every time it receives the detection coordinates 107 generated at a predetermined sampling period from the touch screen 11. The time stamp is time information held by the system and can be shared by each element of the coordinate correction system 200.

  When the coordinate acquisition unit 205 receives the target coordinate 105 corresponding to the time stamp associated with the detected coordinate 107 from the correction amount calculation unit 203, the coordinate acquisition unit 205 sends the target coordinate 105 to the application execution unit 207. When the coordinate acquisition unit 205 does not receive the target coordinate 105 corresponding to the time stamp associated with the detection coordinate 107 from the correction amount calculation unit 203, the coordinate acquisition unit 205 transmits the detection coordinate 107 associated with the time stamp to the application execution unit 207.

  The application execution unit 207 includes a document input program that displays a software keyboard on the touch screen 11, a Web browser that displays an image in which a hyperlink is embedded, an OS interface program that displays an icon on a home screen, and the like. The application execution unit 207 further includes a registration program for registering data necessary for correction in the registration unit 204 through the correction amount calculation unit 203.

  Each program of the application execution unit 207 recognizes the coordinates of the object displayed on the touch screen 11. The application execution unit 207 outputs image data to be displayed on the touch screen 11 to the image data output unit 209. The application execution unit 207 receives the detected coordinate 107 or the target coordinate 105 corresponding to the touch operation performed on the object of the touch screen 11 displayed by the application execution unit 207 from the coordinate acquisition unit 205 and performs an operation corresponding thereto. The image data output unit 209 converts the image data received from the application execution unit 207 into display format data and outputs the data to the touch screen 11.

  Next, the operation of the coordinate correction system 200 will be described with reference to the flowchart of FIG. In block 301, the application execution unit 207 displays an image including an object to be touched on the touch screen 11. The application execution unit 207 prompts the user to direct the surface of the nail 151 toward the camera module 13 in advance to photograph the nail 151 from the front, and registers the nail length s and nail width n measured with high accuracy. 204 may be registered.

  In block 303, the finger 150 contacts the touch surface 11a, and the touch operation is started. The touch operation includes a tap operation for releasing the finger 150 at the touched position and a slide operation for moving the finger while maintaining the touched state. The coordinate correction system 200 can correct the detected coordinates 107 generated by the touch screen 11 along the locus during a slide operation. The touch screen 11 that has detected the contact of the finger 150 starts outputting the detected coordinates 107 at a predetermined sampling period in block 305. Each time the detection coordinate 107 is received, the coordinate acquisition unit 205 adds a time stamp to the received detection coordinate 107 and sends it to the image data acquisition unit 201. The image data acquisition unit 201 that has received the detected coordinates 107 instructs the camera module 13 to transfer image data in block 307.

  Upon receiving the instruction, the camera module 13 starts transferring image data obtained by photographing the user's face and the finger 150 at a predetermined frame rate. The frame rate of the camera module 13 and the transfer cycle of the detection coordinates 107 of the touch screen 11 do not need to be synchronized, but the image data acquisition unit 201 is the block 309 and has a time closest to the time stamp of the detection coordinates 107. Correction data obtained by combining the acquired image data and the detected coordinates 107 together with a time stamp is sent to the correction amount calculation unit 203. Upon receipt of the correction data, the correction amount calculation unit 203 registers the nail size such as the nail length s and nail width n of the finger 150 of the user who is currently touching using the face recognition function in block 311. It is determined whether the unit 204 is registered. When the smartphone 10 is used only by the same user, this procedure may be omitted.

  If registered, the process proceeds to block 317, and if not registered, the process proceeds to block 313. The correction amount calculation unit 203 determines whether the nail size can be calculated from the image data by using the distance d calculated from the detected coordinates 107 in block 313 and the object recognition function. The correction amount calculation unit 203 cannot accurately calculate the nail length s from an image obtained by photographing the finger 150 in a state where the pitch angle β is small from the y-axis direction. Also, the nail width n cannot be accurately calculated from an image taken from the x-axis direction.

  If the nail size cannot be calculated, the process moves to block 351. When the coordinate acquisition unit 205 does not receive the target coordinate 105 corresponding to the time stamp associated with the detected coordinate 107 from the correction amount calculation unit 203 at block 351, the coordinate acquisition unit 205 sends the detected coordinate 107 to the application execution unit 207 at block 351. In this case, the touch operation is processed in a state where the effects of the present invention cannot be obtained, but the operation is not interrupted due to the fact that the correction is not performed. If the nail size can be calculated, the correction amount calculation unit 203 registers the nail size calculated in the registration unit 204 in block 315 and proceeds to block 317.

  In block 317, the correction amount calculation unit 203 uses the object recognition function to determine whether the pitch angle β can be calculated from the image data. When the camera module 13 is photographing the finger 150 from the y-axis direction, the pitch angle β may not be accurately calculated. If the pitch angle β can be calculated with a predetermined accuracy, the process proceeds to block 319, and if not, the process proceeds to block 353. When the coordinate acquisition unit 205 does not receive the target coordinate 105 corresponding to the time stamp associated with the predetermined detection coordinate 107 in block 353, the coordinate acquisition unit 205 sends the detection coordinate 107 to the application execution unit 207.

  In block 319, the correction amount calculation unit 203 uses the object recognition function to determine whether the roll angle γ can be calculated from the image data. When the camera module 13 is photographing the finger 150 from the x-axis direction, the roll angle γ may not be accurately calculated. If the roll angle γ can be calculated with a predetermined accuracy, the process proceeds to block 321; otherwise, the process proceeds to block 353.

  In block 321, the correction amount calculation unit 203 calculates the finger height h <b> 1 using the object recognition function. The correction amount calculation unit 203 calculates offsets Δx and Δy in block 323 by substituting the nail length s, the finger height h1, the pitch angle β, and the roll angle γ into equations (1) and (2). The correction amount calculation unit 203 further calculates the target coordinate 105 by substituting the yaw angle α calculated from the image data into the equations (3) and (4).

  The correction amount calculation unit 203 sends the target coordinates 105 to which the time stamp is added in the block 325 to the coordinate acquisition unit 205. The coordinate acquisition unit 205 sends the target coordinate 105 to the application execution unit 207 instead of the detection coordinate 107 of the same time stamp that has already been acquired. In the case of a slide operation, the procedure returns from block 327 to block 305 and the procedure from block 305 is repeated while the touch screen 13 outputs the detected coordinates 107. When a multi-touch operation is performed as shown in FIG. 1B, the target coordinates 105 can be calculated for each finger by the above procedure.

  Next, another method in which the coordinate correction system 200 corrects the detected coordinates 107 will be described. In the procedure of FIG. 8, the line-of-sight target 155 is set at the center of the nail, and the position of the detection coordinate 107 is related to the finger feature as shown in FIGS. Target coordinates 105 that are satisfactory on average can be generated. However, since the relationship between the detected coordinates 107 according to the shape of the user's finger and the characteristic portion of the finger, the degree of deformation of the finger due to the strength of the pressing force, the position of the line-of-sight target 155, and the like vary from user to user, the results are not necessarily satisfactory. There are also users who cannot obtain or who seek the target coordinates 105 with higher accuracy.

  In addition to the accuracy of the pitch angle β and roll angle γ calculated from the image data, the cause of the error in the offsets Δy and Δx calculated by the equations (1) and (2) is the nail length s and the finger height. The calculation of h1 is not appropriate for the user. FIG. 9 is a coordinate correction for correcting without calculating the nail length s and the finger height h1 from the image data, or assuming the relationship between the position of the line-of-sight target 155 and the detected coordinates 107 and the characteristic part of the finger. 3 is a flowchart illustrating the operation of the system 200.

Here, the registration program constituting the application execution unit 207 estimates the coefficients p and q in the equations (5) and (6) with values calculated from the touch operation for calibration performed by the user.
Δy = p × cos β (5)
Δx = q × tan γ (6)

  In block 401, the registration program of the application execution unit 207 operates to instruct the correction amount calculation unit 203 and the coordinate acquisition unit 205 to perform the registration operation. In block 403, the application execution unit 207 displays the cross line 501a shown in FIG. 10 at a predetermined target coordinate 105 of the touch screen 11, and further performs a touch operation targeting the cross line 501a with a predetermined finger. Prompt to show.

  Since the coefficients p and q change depending on the pressure of the finger during the touch operation, it is desirable that the prompt includes a message that prompts the user to perform the touch operation with the force of the normal operation. In block 405, the user performs a touch operation targeting the cross line 501a. At this time, the user may not mark the line-of-sight target 155 shown in FIG. 4, but the procedure of FIG. 9 is effective for such a user. The procedure of FIG. 9 is also effective for a user whose relationship between the characteristic part of the finger and the detected coordinates 107 is different from the relationship shown in FIGS.

  In block 407, the application execution unit 207 transfers the target coordinates 105, which are the coordinates of the cross line 501a, to the correction amount calculation unit 203. In block 409, the touch screen 11 sends the detected coordinates 107 to the coordinate acquisition unit 205 in response to a touch operation targeting the cross line 501a. When the coordinate acquisition unit 205 adds a time stamp to the detected coordinates 107 and sends it to the image data acquisition unit 201, the camera module 13 starts to transfer the image data.

  In block 411, the correction amount calculation unit 203 calculates the yaw angle α, pitch angle β, and roll angle γ from the image data, and calculates the offsets Δx and Δy from the detected coordinates 107 and the target coordinates 105. In block 413, the correction amount calculation unit 203 calculates the coefficients p and q by substituting the pitch angle β, roll angle γ, and offsets Δx and Δy into equations (5) and (6). Subsequently, the correction amount calculation unit 203 registers the coefficients p and q in the registration unit 204 together with the finger identifier. Upon receiving the registration end notification from the correction amount calculation unit 203 in block 415, the application execution unit 207 returns to block 403, displays the cross line 501b, and repeats the same procedure.

  When registration for all the cross lines 501a to 501f is completed, in block 417, the application execution unit 207 returns to block 403, displays a prompt indicating the next finger, and repeats the same procedure. When the registration of all the fingers is completed, the application execution unit 207 calculates the average value or median value of the six coefficients p and q for each finger in block 419 and registers the registration unit together with the identifier indicating the user's face and the finger identifier. 204. When the registration process is completed in block 421, the correction amount calculation unit 203 for each subsequent detection coordinate 107, the pitch angle β indicating the posture of the finger calculated from the image data, the roll angle γ, and the coefficient registered in the registration unit 204 By substituting the average value or median value of p and q into the equations (5) and (6), the offsets Δx and Δy can be calculated.

  When the correction amount calculation unit 203 registers the coefficients p and q in the procedure of FIG. 9 and uses the equations (5) and (6), the procedure of blocks 311 to 315 and 321 in FIG. 8 can be omitted. . In Expressions (5) and (6), it is not necessary to assume the position of the line-of-sight target 155 and the relationship between the detection coordinates 107 and the characteristic part of the finger. In equations (5) and (6), the two relationships assumed when using equations (1) and (2) do not appear on the surface, but the coefficients p and q calculated in the registration procedure of FIG. Is included.

  Although the present invention has been described with the specific embodiments shown in the drawings, the present invention is not limited to the embodiments shown in the drawings, and is known so far as long as the effects of the present invention are achieved. It goes without saying that any configuration can be adopted.

10 Smartphone 11 Touch screen 11a Touch surface 105 Target coordinate 107 Detection coordinate 109 Contact area 111 Icon 150 Finger 151 Nail 152 Centerline 154 of the length of the nail A line 155 passing through the bottom of the nail and parallel to the touch surface 155 Line-of-sight target 157 Nail Bottom 160 finger axis 162 finger width center 164 finger width center 168 nail width center α yaw angle β pitch angle γ roll angle w1, w2 finger width s nail length n nail width

Claims (18)

For electronic devices equipped with a touch screen that detects coordinates from the contact area of the finger against the touch surface,
Generating detection coordinates in response to a touch operation on the touch screen;
Photographing a fingernail performing the touch operation with a camera;
Identifying the position of the line-of-sight target set at a predetermined position of the nail from a captured image of the camera;
Setting target coordinates at a position associated with the line-of-sight target;
A computer program for executing a process comprising: correcting the detected coordinates with a correction amount calculated from the target coordinates and the detected coordinates . The computer program according to claim 1 for setting the target coordinates directly below the line of sight targets. The computer program according to claim 1 , wherein the line-of-sight target is set on a center line of a finger width formed by projection of the finger on the touch surface and closer to the fingertip side than the bottom of the nail. The computer program according to claim 1 , wherein the step of correcting the detected coordinates includes a step of assuming the detected coordinates immediately below a characteristic part of a finger included in the captured image. The computer program according to claim 1 , wherein the step of correcting the detected coordinates includes a step of calculating a pitch angle formed by a finger axis specified from the captured image and the touch surface. The computer program according to claim 1 , wherein the step of identifying the position of the line-of-sight target includes a step of calculating a distance between the camera and the detection coordinates to calculate a length of the nail. The computer program according to claim 1 , wherein the step of correcting the detected coordinates includes a step of calculating a roll angle around a finger axis specified from the captured image. The computer program according to claim 1 , wherein the step of correcting the detected coordinates includes a step of calculating a height of a finger with respect to the touch surface specified from the captured image by calculating a distance between the camera and the detected coordinates. . For electronic devices equipped with a touch screen that detects coordinates from the contact area of the finger against the touch surface,
Displaying a target object on the touch screen;
Said fingers the target coordinates of the target object touch screen corresponding to the difference between the detected coordinates of detecting an offset in the longitudinal direction and [Delta] y, the pitch angle of the finger for the touch operation and beta, and the coefficient p Formula Preparing (a);
Δy = p × cos β (a)
Generating detection coordinates in response to a touch operation on the target object ;
Photographing a finger performing the touch operation with a camera;
Substituting the pitch angle β calculated based on the finger posture specified from the photographed image of the finger and the offset Δy calculated from the target coordinates and the detected coordinates into the equation (a), and calculating the coefficient p; A computer program for executing a process having: Formula (b) is prepared in which the offset in the width direction of the finger corresponding to the difference between the target coordinates and the detected coordinates detected by the touch screen is Δx, the roll angle of the finger performing the touch operation is γ, and the coefficient is q. And steps to
Δx = q × tan γ (b)
  Substituting the roll angle γ calculated based on the finger posture specified from the photographed image of the finger and the offset Δx calculated from the target coordinates and the detected coordinates into the equation (b), and calculating the coefficient q;
The computer program according to claim 9. The recording medium which recorded the computer program in any one of Claims 1-10. The electronic device carrying the recording medium of Claim 11. A touch screen that detects coordinates from the contact area of the finger with respect to the touch surface and outputs detected coordinates;
A camera capable of shooting a finger performing a touch operation and outputting image data;
The image data identifies the position of the gaze target set at a predetermined position of the nail from, and set the target coordinates to the location associated with the visual axis target, calculating the detected coordinates or al correction amount and the target coordinate correction amount An electronic device having a calculation unit. The electronic device according to claim 13, wherein the touch screen is a capacitance type .   The electronic device according to claim 13, wherein the camera is a wide-angle lens camera or a fish-eye lens camera.   The electronic apparatus according to claim 15, wherein only one camera is mounted on any frame edge around the touch screen. The electronic apparatus according to claim 13, further comprising: a coordinate acquisition unit that outputs the detected coordinates to an application execution unit when the correction amount calculation unit does not calculate the correction amount . An electronic device equipped with a camera and a touch screen that outputs detection coordinates from a contact area of a finger with respect to a touch surface is a method of correcting the detection coordinates,
Receiving detected coordinates in response to a touch operation on the touch screen;
Receiving image data from the camera that captured the fingernail performing the touch operation;
Identifying the position of the line-of-sight target set at a predetermined position of the nail from the image data;
Setting target coordinates at a position associated with the line-of-sight target;
Calculating a correction amount from the target coordinates and the detected coordinates ;
Having a method.

Images