JP6368455B2 - Apparatus, method, and program - Google Patents

Description

  The present application relates to an apparatus, a method, and a program. In particular, the present application relates to a device having a touch screen, a method for controlling the device, and a program for controlling the device.

  Touch screen devices comprising a touch screen are known. Touch screen devices include, but are not limited to, for example, smartphones and tablets. The touch screen device detects a finger, pen, or stylus pen gesture via the touch screen. The touch screen device operates according to the detected gesture. An example of the operation according to the detected gesture is described in Patent Document 1, for example.

  The basic operation of the touch screen device is realized by an OS (Operating System) installed in the device. Examples of the OS installed in the touch screen device include, but are not limited to, Android (registered trademark), BlackBerry (registered trademark) OS, iOS, Symbian (registered trademark) OS, and Windows (registered trademark) Phone.

International Publication No. 2008/086302

  Touch screen devices are said to provide users with intuitive operability. However, depending on the screen displayed by the touch screen device, it may be difficult for the user to understand what kind of gesture is performed and what processing is executed.

  From the above, there is a need for an apparatus, a method, and a program that allow a user to easily know what gesture is assigned to a displayed screen.

  An apparatus according to an aspect includes a display that displays a screen to which a gesture is assigned, a touch screen that detects the gesture, and a gesture that is detected when the gesture is detected while the screen is displayed. A controller that executes corresponding processing. The controller further displays information on the gesture on the display when the screen is displayed on the display.

  A method according to one aspect is a method of controlling a device having a display, the step of displaying a screen to which a gesture is assigned on the display, and information about the gesture when the screen is displayed. A step of further displaying on the display; and a step of executing processing corresponding to the gesture when the gesture is detected while the screen is displayed.

  A program according to one aspect includes a step of displaying a screen to which a gesture is assigned on an apparatus having a display on the display, and a step of further displaying information regarding the gesture on the display when the screen is displayed. And executing a process corresponding to the gesture when the gesture is detected while the screen is displayed.

FIG. 1 is a perspective view of the smartphone according to the embodiment. FIG. 2 is a front view of the smartphone. FIG. 3 is a rear view of the smartphone. FIG. 4 is a diagram illustrating an example of the home screen. FIG. 5 is a block diagram of the smartphone. FIG. 6 is a diagram illustrating an example of a screen displayed on the display. FIG. 7 is a diagram illustrating an example in which a gesture assigned to a screen is presented to a user. FIG. 8 is a diagram illustrating an example in which information related to processing executed in response to a gesture is included in the operation guide. FIG. 9 is a diagram illustrating an example in which an operation guide is displayed at the center of the screen. FIG. 10 is a diagram illustrating an example in which an operation guide is displayed at the top of the screen. FIG. 11 is a diagram illustrating an example of a method for creating operation guide information. FIG. 12 is a diagram illustrating an example of the screen definition file. FIG. 13 is a diagram illustrating an example of creating operation guide information based on an operation history. FIG. 14 is a flowchart illustrating an example of a processing procedure for displaying an operation guide.

  Embodiments for carrying out the present invention will be described in detail with reference to the drawings. Below, a smart phone is demonstrated as an example of an apparatus provided with a touch screen.

(Embodiment)
The overall configuration of the smartphone 1 according to the embodiment will be described with reference to FIGS. 1 to 3. As shown in FIGS. 1 to 3, the smartphone 1 has a housing 20. The housing 20 includes a front face 1A, a back face 1B, and side faces 1C1 to 1C4. The front face 1 </ b> A is the front of the housing 20. The back face 1 </ b> B is the back surface of the housing 20. The side faces 1C1 to 1C4 are side surfaces that connect the front face 1A and the back face 1B. Hereinafter, the side faces 1C1 to 1C4 may be collectively referred to as the side face 1C without specifying which face.

  The smartphone 1 includes a touch screen display 2, buttons 3A to 3C, an illuminance sensor 4, a proximity sensor 5, a receiver 7, a microphone 8, and a camera 12 on the front face 1A. The smartphone 1 has a speaker 11 and a camera 13 on the back face 1B. The smartphone 1 has buttons 3D to 3F and a connector 14 on the side face 1C. Hereinafter, the buttons 3A to 3F may be collectively referred to as the button 3 without specifying which button.

  The touch screen display 2 includes a display 2A and a touch screen 2B. In the example of FIG. 1, the display 2A and the touch screen 2B are substantially rectangular, but the shapes of the display 2A and the touch screen 2B are not limited to this. Each of the display 2A and the touch screen 2B can take any shape such as a square or a circle. In the example of FIG. 1, the display 2 </ b> A and the touch screen 2 </ b> B are overlapped, but the arrangement of the display 2 </ b> A and the touch screen 2 </ b> B is not limited to this. For example, the display 2A and the touch screen 2B may be arranged side by side or may be arranged apart from each other. In the example of FIG. 1, the long side of the display 2A is along the long side of the touch screen 2B, and the short side of the display 2A is along the short side of the touch screen 2B, but the display 2A and the touch screen 2B are overlapped. Is not limited to this. When the display 2A and the touch screen 2B are arranged so as to overlap each other, for example, one or more sides of the display 2A may not be along any side of the touch screen 2B.

  The display 2A includes a liquid crystal display (LCD: Liquid Crystal Display), an organic EL display (OELD: Organic Electro-Luminescence Display), or an inorganic EL display (IELD: Inorganic Electro-Luminescence Display). The display 2A displays characters, images, symbols, graphics, and the like.

  The touch screen 2B detects contact of a finger, a pen, a stylus pen, or the like with respect to the touch screen 2B. The touch screen 2B can detect a position where a plurality of fingers, a pen, a stylus pen, or the like is in contact with the touch screen 2B. In the following description, a finger, pen, stylus pen, or the like that contacts the touch screen 2B may be referred to as a “contact object (contact object)”.

  The detection method of the touch screen 2B may be any method such as a capacitance method, a resistive film method, a surface acoustic wave method (or an ultrasonic method), an infrared method, an electromagnetic induction method, and a load detection method. In the following description, in order to simplify the description, it is assumed that the user uses the finger to touch the touch screen 2B in order to operate the smartphone 1.

  The smartphone 1 is based on at least one of the contact detected by the touch screen 2B, the position at which the contact is detected, the change in the position at which the contact is detected, the interval at which the contact is detected, and the number of times the contact is detected. Determine the type of gesture. The gesture is an operation performed on the touch screen 2B. Gestures identified by the smartphone 1 include, but are not limited to, touch, long touch, release, swipe, tap, double tap, long tap, drag, flick, pinch in, and pinch out, for example.

  “Touch” is a gesture in which a finger touches the touch screen 2B. The smartphone 1 determines a gesture in which a finger contacts the touch screen 2B as a touch. “Long touch” is a gesture in which a finger touches the touch screen 2B for a longer period of time. The smartphone 1 determines a gesture in which a finger contacts the touch screen 2B for longer than a certain time as a long touch.

  “Release” is a gesture in which a finger leaves the touch screen 2B. The smartphone 1 determines that a gesture in which a finger leaves the touch screen 2B is a release. “Swipe” is a gesture in which a finger moves while touching the touch screen 2B. The smartphone 1 determines a gesture that moves while the finger is in contact with the touch screen 2B as a swipe.

  A “tap” is a gesture for releasing following a touch. The smartphone 1 determines a gesture for releasing following a touch as a tap. The “double tap” is a gesture in which a gesture for releasing following a touch is continued twice. The smartphone 1 determines a gesture in which a gesture for releasing following a touch is continued twice as a double tap.

  “Long tap” is a gesture for releasing following a long touch. The smartphone 1 determines a gesture for releasing following a long touch as a long tap. “Drag” is a gesture for performing a swipe starting from an area where a movable object is displayed. The smartphone 1 determines, as a drag, a gesture for performing a swipe starting from an area where a movable object is displayed.

  “Flick” is a gesture in which a finger leaves the touch screen 2B while moving after touching the touch screen 2B. In other words, “flick” is a gesture in which a release is performed while a finger moves following a touch. The smartphone 1 determines, as a flick, a gesture in which the finger leaves the touch screen 2B while moving after touching the touch screen 2B. The flick is often performed while the finger moves in one direction. Flick is "upper flick" where the finger moves upward on the screen, "lower flick" where the finger moves downward on the screen, "right flick" where the finger moves rightward on the screen, finger is left on the screen Including “left flick” moving in the direction. The movement of a finger in a flick is often quicker than the movement of a finger in a swipe.

  “Pinch-in” is a gesture of swiping in a direction in which a plurality of fingers approach each other. The smartphone 1 determines, as a pinch-in, a gesture in which the distance between the position of a finger detected by the touch screen 2B and the position of another finger is shortened. “Pinch out” is a gesture of swiping a plurality of fingers away from each other. The smartphone 1 determines, as a pinch-out, a gesture that increases the distance between the position of one finger and the position of another finger detected by the touch screen 2B.

  In the present embodiment, a gesture performed with one finger may be referred to as a “single touch gesture”. In the present embodiment, a gesture performed with two or more fingers may be referred to as a “multi-touch gesture”. Pinch-in and pinch-out are multi-touch gestures. Taps, flicks, swipes, and the like are single-touch gestures when performed with one finger, and multi-touch gestures when performed with two or more fingers.

  The smartphone 1 operates according to these gestures that are determined via the touch screen 2B. Therefore, an operability that is intuitive and easy to use for the user is realized. The operation performed by the smartphone 1 according to the determined gesture may differ depending on the screen displayed on the display 2A. In the following description, in order to simplify the description, “the touch screen 2B detects a gesture and the smartphone 1 determines that the type of the detected gesture is X” and “the smartphone 1 detects X”. Or “the controller detects X”.

  An example of a screen displayed on the display 2A will be described with reference to FIG. FIG. 4 shows an example of the home screen. The home screen is sometimes called a desktop, a standby screen, an idle screen, or a standard screen. The home screen is displayed on the display 2A. The home screen is a screen that allows the user to select which application to execute from among the applications installed in the smartphone 1. The smartphone 1 executes the application selected on the home screen in the foreground. The screen of the application executed in the foreground is displayed on the display 2A.

  The smartphone 1 can place an icon on the home screen. A plurality of icons 50 are arranged on the home screen 40 shown in FIG. Each icon 50 is associated with an application installed in the smartphone 1 in advance. When the smartphone 1 detects a gesture for the icon 50, the smartphone 1 executes an application associated with the icon 50 from which the gesture is detected. For example, the smartphone 1 executes the mail application when a tap on the icon 50 associated with the mail application is detected.

  The icon 50 includes an image and a character string. The icon 50 may include a symbol or a graphic instead of the image. The icon 50 may not include either an image or a character string. The icons 50 are arranged based on the arrangement pattern. A wallpaper 41 is displayed behind the icon 50. The wallpaper is sometimes called a photo screen, a back screen, an idle image or a background image. The smartphone 1 can use any image as the wallpaper 41. The smartphone 1 may be configured so that the user can select an image to be displayed as the wallpaper 41.

  The smartphone 1 may have a plurality of home screens. For example, the smartphone 1 determines the number of home screens according to the setting by the user. The smartphone 1 displays the selected one on the display 2A even if there are a plurality of home screens.

  The smartphone 1 can change the home screen displayed on the display 2A. When a gesture is detected while displaying one of the home screens, the smartphone 1 changes the home screen displayed on the display 2A to another home screen. For example, when detecting a right flick, the smartphone 1 changes the home screen displayed on the display 2A to the left home screen. For example, when the smartphone 1 detects a left flick, the smartphone 1 changes the home screen displayed on the display 2A to the right home screen. When the gesture is detected while displaying the first home screen on the display 2A, the smartphone 1 gradually reduces the area of the first home screen displayed on the display 2A, and the second displayed on the display 2A. The home screen displayed on the display 2A is changed from the first home screen to the second home screen so that the area of the home screen gradually increases. The smartphone 1 may switch the home screen so that the first home screen is immediately replaced with the second home screen.

  A region 42 is provided at the upper end of the display 2A. In the area 42, a remaining amount mark 43 indicating the remaining amount of the power source and a radio wave level mark 44 indicating the electric field strength of the communication radio wave are displayed. The smartphone 1 may display the time, weather, running application, communication system type, telephone status, device mode, event occurring in the device, and the like in the area 42. As described above, the area 42 is used for various notifications to the user. The area 42 may be provided on a screen other than the home screen 40. The position where the region 42 is provided is not limited to the upper end of the display 2A.

  The home screen 40 shown in FIG. 4 is an example, and the form of various elements, the arrangement of various elements, the number of home screens 40, how to perform various operations on the home screen 40, and the like are as described above. Not necessarily.

  FIG. 5 is a block diagram of the smartphone 1. The smartphone 1 includes a touch screen display 2, a button 3, an illuminance sensor 4, a proximity sensor 5, a communication unit 6, a receiver 7, a microphone 8, a storage 9, a controller 10, a speaker 11, and a camera. 12 and 13, a connector 14, an acceleration sensor 15, an orientation sensor 16, and a gyroscope 17.

  As described above, the touch screen display 2 includes the display 2A and the touch screen 2B. The display 2A displays characters, images, symbols, graphics, or the like. The touch screen 2B detects a gesture.

  The button 3 is operated by the user. The button 3 includes buttons 3A to 3F. The controller 10 detects an operation on the button 3 by cooperating with the button 3. The operation on the button 3 includes, for example, click, double click, triple click, push, and multi-push, but is not limited thereto.

  The buttons 3A to 3C are, for example, a home button, a back button, or a menu button. The button 3D is, for example, a power on / off button of the smartphone 1. The button 3D may also serve as a sleep / sleep release button. The buttons 3E and 3F are volume buttons, for example.

  The illuminance sensor 4 detects the illuminance of the ambient light of the smartphone 1. Illuminance indicates light intensity, brightness, or luminance. The illuminance sensor 4 is used for adjusting the luminance of the display 2A, for example. The proximity sensor 5 detects the presence of a nearby object without contact. The proximity sensor 5 detects the presence of an object based on a change in a magnetic field or a change in a feedback time of an ultrasonic reflected wave. The proximity sensor 5 detects that the touch screen display 2 is brought close to the face, for example. The illuminance sensor 4 and the proximity sensor 5 may be configured as one sensor. The illuminance sensor 4 may be used as a proximity sensor.

  The communication unit 6 communicates wirelessly. The communication method supported by the communication unit 6 is a wireless communication standard. Examples of wireless communication standards include cellular phone communication standards such as 2G, 3G, and 4G. As cellular phone communication standards, for example, LTE (Long Term Evolution), W-CDMA (Wideband Code Multiple Access), CDMA2000, PDC (Personal Digital Cellular, GSM (registered trademark) mmloS). (Personal Handy-phone System). As wireless communication standards, for example, there are WiMAX (Worldwide Interoperability for Microwave Access), IEEE802.11, Bluetooth (registered trademark), IrDA (Infrared Data Association), NFC (NearCo), etc. The communication unit 6 may support one or more of the communication standards described above.

  The receiver 7 and the speaker 11 are sound output units. The receiver 7 and the speaker 11 output the sound signal transmitted from the controller 10 as sound. The receiver 7 is used, for example, to output the other party's voice during a call. The speaker 11 is used for outputting a ring tone and music, for example. One of the receiver 7 and the speaker 11 may also function as the other. The microphone 8 is a sound input unit. The microphone 8 converts the user's voice or the like into a sound signal and transmits the sound signal to the controller 10.

  The storage 9 stores programs and data. The storage 9 is also used as a work area for temporarily storing the processing result of the controller 10. The storage 9 may include any non-transitory storage medium such as a semiconductor storage medium and a magnetic storage medium. The storage 9 may include a plurality of types of storage media. The storage 9 may include a combination of a portable storage medium such as a memory card, an optical disk, or a magneto-optical disk and a storage medium reader. The storage 9 may include a storage device used as a temporary storage area such as a RAM (Random Access Memory).

  The programs stored in the storage 9 include an application executed in the foreground or the background and a control program that supports the operation of the application. For example, the application displays a screen on the display 2 </ b> A, and causes the controller 10 to execute a process according to a gesture detected by the touch screen 2 </ b> B. The control program is, for example, an OS. The application and the control program may be installed in the storage 9 via wireless communication by the communication unit 6 or a non-transitory storage medium.

  The storage 9 stores, for example, a control program 9A, a mail application 9B, a browser application 9C, an SNS (Social Networking Service) application 9D, operation guide information 9Y, and setting data 9Z. The mail application 9B provides an electronic mail function for creating, transmitting, receiving, and displaying an electronic mail. The browser application 9C provides a WEB browsing function for displaying a WEB page. The SNS application 9D provides an access function to the SNS including a message service, a diary service, a community service, and the like. The operation guide information 9Y includes information regarding an operation (gesture) assigned to each screen displayed on the display 2A. The setting data 9Z includes information related to various settings related to the operation of the smartphone 1.

  The control program 9A provides functions related to various controls for operating the smartphone 1. The control program 9A realizes a call by controlling the communication unit 6, the receiver 7, the microphone 8, and the like, for example. The functions provided by the control program 9A include a function of presenting information related to gestures assigned to the screen displayed on the display 2A to the user. The function provided by the control program 9A may be used in combination with a function provided by another program such as the mail application 9B.

  The controller 10 is an arithmetic processing device. The arithmetic processing unit includes, for example, a CPU (Central Processing Unit), an SoC (System-on-a-chip), an MCU (Micro Control Unit), and an FPGA (Field-Programmable Gate Array), but is not limited thereto. The controller 10 controls various operations of the smartphone 1 to realize various functions.

  Specifically, the controller 10 executes instructions included in the program stored in the storage 9 while referring to the data stored in the storage 9 as necessary. And the controller 10 controls a function part according to data and a command, and implement | achieves various functions by it. The functional unit includes, for example, the display 2A, the communication unit 6, the microphone 8, and the speaker 11, but is not limited thereto. The controller 10 may change the control according to the detection result of the detection unit. Examples of the detection unit include, but are not limited to, the touch screen 2B, the button 3, the illuminance sensor 4, the proximity sensor 5, the receiver 7, the camera 12, the camera 13, the acceleration sensor 15, the orientation sensor 16, and the gyroscope 17. .

  For example, the controller 10 executes the control program 9A to execute various controls such as a control for presenting information about a gesture assigned to the screen displayed on the display 2A to the user.

  The camera 12 is an in-camera that captures an object facing the front face 1A. The camera 13 is an out camera that captures an object facing the back face 1B.

  The connector 14 is a terminal to which another device is connected. The connector 14 may be a general-purpose terminal such as a universal serial bus (USB), a high-definition multimedia interface (HDMI) (registered trademark), a light peak (thunderbolt (registered trademark)), or an earphone microphone connector. . The connector 14 may be a dedicated terminal such as a dock connector. Devices connected to the connector 14 include, but are not limited to, external storage, speakers, and communication devices, for example.

  The acceleration sensor 15 detects the direction and magnitude of acceleration acting on the smartphone 1. The direction sensor 16 detects the direction of geomagnetism. The gyroscope 17 detects the angle and angular velocity of the smartphone 1. The detection results of the acceleration sensor 15, the azimuth sensor 16, and the gyroscope 17 are used in combination in order to detect changes in the position and orientation of the smartphone 1.

  In FIG. 5, some or all of the programs and data stored in the storage 9 may be downloaded from other devices by wireless communication using the communication unit 6. In FIG. 5, some or all of the programs and data stored in the storage 9 may be stored in a non-transitory storage medium that can be read by the reading device included in the storage 9. 5 may be stored in a non-transitory storage medium that can be read by a reading device connected to the connector 14. Non-transitory storage media include, for example, optical disks such as CD (registered trademark), DVD (registered trademark), and Blu-ray (registered trademark), magneto-optical disks, magnetic storage media, memory cards, and solid-state storage media Including, but not limited to.

  The configuration of the smartphone 1 shown in FIG. 5 is an example, and may be appropriately changed within a range not impairing the gist of the present invention. For example, the number and type of buttons 3 are not limited to the example of FIG. The smartphone 1 may include buttons such as a numeric keypad layout or a QWERTY layout as buttons for operations related to the screen instead of the buttons 3A to 3C. The smartphone 1 may include only one button or may not include a button for operations related to the screen. In the example illustrated in FIG. 5, the smartphone 1 includes two cameras, but the smartphone 1 may include only one camera or may not include a camera. In the example illustrated in FIG. 5, the smartphone 1 includes three types of sensors in order to detect the position and orientation, but the smartphone 1 may not include some of these sensors. Alternatively, the smartphone 1 may include another type of sensor for detecting at least one of a position and a posture.

  Next, control for presenting information related to gestures assigned to the screen to the user will be described with reference to FIGS. FIG. 6 is a diagram illustrating an example of a screen displayed on the display 2A. FIG. 7 is a diagram illustrating an example in which a gesture assigned to a screen is presented to a user.

  In the example shown in FIG. 6, the display 2 </ b> A displays a diary screen 60. The diary screen 60 is one of the screens provided by the SNS application 9D. The diary screen 60 includes a profile display area 61, a diary display area 62, and a toolbar area 63.

  The profile display area 61 is an area in which the diary author's profile is displayed. The diary display area 62 is an area where a diary is displayed. The diary display area 62 is configured to scroll vertically. The toolbar area 63 is an area where icons for executing predetermined processing are arranged. In the example shown in FIG. 6, the toolbar area 63 includes an icon 63 a for displaying the user's home page, an icon 63 b for displaying a diary search page, and an icon 63 c for displaying a comment input screen. It is out.

  As shown in FIG. 7, the smartphone 1 displays an operation guide on the diary screen 60 in order to present a gesture assigned to the diary screen 60 to the user. In the example shown in FIG. 7, a balloon 61a, a balloon 62a, a balloon 63d, a balloon 63e, and a balloon 63f are displayed as operation guides.

  The balloon 61 a includes a character string “tap” and points to the profile display area 61. That is, the balloon 61a indicates that a tap is assigned to the profile display area 61 as a gesture for executing some processing.

  The balloon 62 a includes character strings “tap”, “long tap”, “flick”, and “swipe”, and points to the diary display area 62. That is, the balloon 62 a indicates that tap, long tap, flick, and swipe are assigned to the diary display area 62 as gestures for executing some processing.

  The balloon 63d includes a character string “tap” and points to the icon 63a. The balloon 63e includes a character string “tap” and points to the icon 63b. The balloon 63f includes a character string “tap” and points to the icon 63c. That is, the balloons 63d to 63f indicate that taps are assigned to the icons 63a to 63c, respectively, as gestures for executing some processing.

  In this way, by displaying the operation guide, the smartphone 1 can cause the user to know what kind of processing is executed by performing what kind of gesture on which part of the screen. If the user can grasp what kind of gesture is assigned to which part of the screen, the user can know the processing to be executed in accordance with the gesture on the screen by knowledge or trial and error regarding a similar screen. .

  The smartphone 1 may include information related to processing executed according to the gesture in the operation guide. FIG. 8 is a diagram illustrating an example in which information related to processing executed in response to a gesture is included in the operation guide. In the example shown in FIG. 8, a balloon 61b, a balloon 62b, a balloon 63g, a balloon 63h, and a balloon 63i are displayed as operation guides.

  The balloon 61b includes a character string “tap: detailed display” and points to the profile display area 61. That is, the balloon 61b indicates that a tap is assigned to the profile display area 61 as a gesture for executing some processing. Furthermore, the balloon 61b indicates that the details of the diary author's profile are displayed when the profile display area 61 is tapped.

  The balloon 62b includes character strings “tap: detail display”, “long tap: menu”, “flick: scroll”, and “swipe: scroll”, and points to the diary display area 62. That is, the balloon 62b indicates that tap, long tap, flick, and swipe are assigned to the diary display area 62 as gestures for executing some processing. Further, the balloon 62b displays details of the diary when a tap is performed on the diary display area 62, a menu is displayed when a long tap is performed on the diary display area 62, and a flick or This shows that the diary scrolls when a swipe is performed.

  The balloon 63g includes a character string “tap: home” and points to the icon 63a. The balloon 63h includes a character string “tap: search” and points to the icon 63b. The balloon 63i includes a character string “tap: comment” and points to the icon 63c. That is, the balloons 63g to 63i indicate that taps are assigned to the icons 63a to 63c, respectively, as gestures for executing some processing. Further, the balloon 63g indicates that the user's home page is displayed when the icon 63a is tapped. The balloon 63h indicates that a diary search page is displayed when the icon 63b is tapped. A balloon 63i indicates that a comment input screen is displayed when the icon 63c is tapped.

  In this way, by including in the operation guide information related to processing executed according to a gesture, the smartphone 1 uses what type of gesture is performed on which part of the screen. Can be made to grasp.

  7 and 8 show examples in which a balloon is displayed as an operation guide. However, the smartphone 1 may display information that allows the user to grasp the type of gesture assigned to the screen as an operation guide. The information that allows the user to grasp the type of gesture is, for example, a character such as a character, an image, a diagram, or an animation representing the gesture. Information that allows the user to grasp the type of gesture may be displayed in an overlapping manner on the area to which the gesture is assigned.

  7 and 8 show an example in which an operation guide is displayed in association with each region to which a gesture is assigned. However, the smart phone 1 should just display an operation guide so that a user can grasp | ascertain the gesture allocated somewhere on the screen currently displayed on the display 2A at least.

  FIG. 9 is a diagram illustrating an example in which an operation guide is displayed at the center of the screen. In the example shown in FIG. 9, a symbol 60a representing a tap, a symbol 60b representing a long tap, a symbol 60c representing a flick, and a symbol 60d representing a swipe are included in the diary. Overlaid on the screen 60.

  The symbol 60a indicates that a tap is assigned somewhere on the diary screen 60 as a gesture for executing some processing. The symbol 60b indicates that a long tap is assigned somewhere on the diary screen 60 as a gesture for executing some processing. The symbol 60c indicates that a flick is assigned somewhere on the diary screen 60 as a gesture for executing some processing. The symbol 60d indicates that a swipe is assigned somewhere on the diary screen 60 as a gesture for executing some processing.

  As described above, the smartphone 1 can make the user grasp the gesture assigned to the screen without displaying the operation guide in association with the area to which the gesture is assigned.

  In the example shown in FIG. 9, the first letter of the gesture is displayed as information representing the gesture. However, the symbol representing the gesture is not limited to this. For example, an image, a diagram, or an animation representing a gesture may be displayed as information representing the gesture.

  In the example shown in FIG. 9, an operation guide is displayed at the center of the screen. However, the position where the operation guide is displayed is not limited to this. For example, the smartphone 1 may display an operation guide at a position that does not prevent the user from seeing elements on the screen.

  FIG. 10 is a diagram illustrating an example in which an operation guide is displayed at a position that does not prevent the user from seeing elements on the screen. In the example shown in FIG. 10, an operation guide area 60 e is displayed at the top of the diary screen 60. The operation guide area 60e includes character strings of “tap”, “long tap”, “flick”, and “swipe”. These character strings indicate that taps, long taps, flicks, and swipes are assigned somewhere on the diary screen 60 as gestures for executing some processing.

  The operation guide area 60e is displayed so as to overlap the title portion of the diary screen 60. Even if the title part is covered with the operation guide area 60e, each element to which the gesture is assigned on the diary screen 60 remains exposed. Thus, by displaying the operation guide at a position that does not overlap with the elements arranged on the screen, it is possible to prevent the display of the operation guide from impairing the visibility of the screen.

  As described above, the smartphone 1 presents information related to the gesture assigned to the screen to the user by displaying the operation guide. By viewing the operation guide, the user can grasp what kind of gesture is assigned to the screen even if it is the first screen to be viewed.

  The timing at which the smartphone 1 displays the operation guide is, for example, the timing at which the screen is displayed, the timing at which no gesture on the screen is detected for longer than a predetermined period, and the timing at which a predetermined instruction from the user is detected. At least one. The predetermined instruction by the user is performed, for example, by at least one of a predetermined gesture on the touch screen 2B, pressing of the button 3, an operation of shaking the smartphone 1, and an input of voice.

  When the operation guide is displayed at the timing when the screen is displayed, the user can view the operation guide on the screen every time a new screen is displayed. If the operation guide is displayed at a timing when the gesture for the screen is not detected longer than the specified period, the operation guide is automatically displayed when the user is confused without knowing the gesture assigned to the screen. Is displayed. When the operation guide is displayed at a timing when a predetermined instruction by the user is detected, the user can display the operation guide at an arbitrary timing.

  The timing at which the smartphone 1 finishes displaying the operation guide includes, for example, the timing at which a gesture assigned to the screen is detected, the timing at which a predetermined period has elapsed after the operation guide is displayed, and the predetermined instruction from the user At least one of the timing when the screen is switched and the timing when the screen is switched. The predetermined instruction by the user is performed, for example, by at least one of a predetermined gesture on the touch screen 2B, pressing of the button 3, an operation of shaking the smartphone 1, and an input of voice.

  When the display of the operation guide ends at the timing when the gesture assigned to the screen is detected, the user simply hides the operation guide without performing an extra operation simply by performing a gesture for executing a desired process. Can be displayed. When the display of the operation guide ends at a timing when a predetermined period has elapsed since the operation guide was displayed, the user does not need to perform any operation in order to hide the operation guide. When the display of the operation guide ends at a timing when a predetermined instruction by the user is detected, the user can hide the operation guide at an arbitrary timing.

  The smartphone 1 may display the operation guide in a translucent manner. In this case, the smartphone 1 may be configured so that the user can set the transparency of the operation guide. Alternatively, the smartphone 1 may display the operation guide in a blinking manner. By controlling the display of the operation guide in this way, the user can grasp what elements the screen includes even when the operation guide is displayed.

  After displaying the operation guide on a certain screen a predetermined number of times, the smartphone 1 may not display the operation guide when displaying the screen. The predetermined number of times is set to the number of times that the user can almost remember the gesture assigned to the screen. The smartphone 1 may be configured so that the user can set a predetermined number of times. Even after the operation guide is displayed a predetermined number of times, the smartphone 1 may display the operation guide when a predetermined instruction from the user is detected.

  The smartphone 1 displays the operation guide based on the operation guide information 9Y. The operation guide information 9Y holds information for identifying a screen and information indicating a gesture assigned to the screen in association with each other. When the gesture is assigned to a specific area on the screen, the operation guide information 9Y may hold information indicating the gesture in further association with information indicating the area to which the gesture is assigned. In FIG. 5, the operation guide information 9Y is illustrated as a single piece of information independent of the application. However, the operation guide information 9Y may be stored for each application or embedded in each application. Also good.

  The operation guide information 9Y may be provided by a creator of an application that provides a screen and may be introduced into the smartphone 1 together with the application.

  As illustrated in FIG. 11, the operation guide information 9 </ b> Y may be created based on a result of performing an analysis process on an application 9 </ b> X that provides a screen. The device that performs the analysis process on the application 9X may be the smartphone 1 or may be a device that is owned by a vendor that distributes the application 9X.

  The analysis process for the application 9X is, for example, a process of determining whether a handler is created for an event corresponding to various gestures based on a code or a character string included in the application 9X. A handler is a collection of codes for realizing a process to be executed when an event is detected. When a handler is created for an event corresponding to a gesture for a certain screen (element on the screen), it can be determined that the gesture is assigned to that screen (element on the screen).

  When the application is configured to read the definition of the screen from the definition file, the operation guide information 9Y may be created based on the definition file. The definition file is, for example, a screen definition file or an HTML (HyperText Markup Language) file. The HTML file may include JavaScript. The definition file may be embedded inside the application or may exist outside the application.

  FIG. 12 is a diagram illustrating an example of the screen definition file. The screen definition file 70 shown in FIG. 12 is configured in an XML (Extensible Markup Language) format so as to display the diary screen 60 shown in FIG. The screen definition file 70 includes a Canvas tag 71 corresponding to the profile display area 61, a ListView tag 72 corresponding to the diary display area 62, and a Bar tag 73 corresponding to the toolbar area 63.

  The Canvas tag 71 includes an onTap attribute 71 a in addition to an attribute that specifies the position and size of the profile display area 61. In the onTap attribute 71a, a name of a handler including a process to be executed when a tap is detected in a corresponding area and a character string representing an outline of the process included in the handler are set. In the example illustrated in FIG. 12, a handler name “showDetails” and a character string “detailed display” are set in the onTap attribute 71 a. According to the onTap attribute 71a, it is determined that a tap is assigned to the profile display area 61 for processing related to “detail display”.

  The ListView tag 72 includes an onTap attribute 72a, an onLongTap attribute 72b, and a scrollable attribute 72c, in addition to attributes that specify the position and size of the diary display area 62. In the onTap attribute 72a, a name of a handler including a process to be executed when a tap is detected in a corresponding area and a character string representing an outline of the process included in the handler are set. In the onLongTap attribute 72b, a name of a handler including a process to be executed when a long tap is detected in a corresponding area and a character string representing an outline of the process included in the handler are set. A value indicating whether scrolling is possible is set in the scrollable attribute 72c.

  In the example illustrated in FIG. 12, a handler name “showDetails” and a character string “detailed display” are set in the onTap attribute 72 a. According to the onTap attribute 72a, it is determined that a tap is assigned to the diary display area 62 for processing related to “detail display”. In the example illustrated in FIG. 12, a handler name “showMenu” and a character string “menu” are set in the onLongTap attribute 72 b. According to the onLongTap attribute 72b, it is determined that a long tap is assigned to the diary display area 62 for processing related to “menu”.

  In the example illustrated in FIG. 12, “true” is set in the scrollable attribute 72 c. According to the scrollable attribute 72c, it is determined that the diary display area 62 is set to be scrollable. That is, according to the scrollable attribute 72c, it is determined that the diary display area 62 is assigned with a flick and swipe that are gestures for scrolling.

  The Bar tag 73 includes an Icon tag 73a corresponding to the icon 63a, an Icon tag 73b corresponding to the icon 63b, and an Icon tag 73c corresponding to the icon 63c. Each of the icons 63a to 63b includes onTap attributes 73d to 73f in addition to an attribute for designating an image displayed as an icon. In each of the onTap attributes 73d to 73f, a name of a handler including a process to be executed when a tap on the corresponding icon is detected, and a character string representing an outline of the process included in the handler are set.

  In the example illustrated in FIG. 12, a handler name “showHome” and a character string “home” are set in the onTap attribute 73 d. According to the onTap attribute 73d, it is determined that a tap is assigned to the icon 63a for processing related to “home”. In the example illustrated in FIG. 12, a handler name “inputQuery” and a character string “search” are set in the onTap attribute 73 e. According to the onTap attribute 73e, it is determined that a tap is assigned to the icon 63b for processing related to “search”. In the example shown in FIG. 12, a handler name “inputComment” and a character string “comment” are set in the onTap attribute 73f. According to the onTap attribute 73f, it is determined that a tap is assigned to the icon 63c for processing related to “comment”.

  When the definition file is embedded in the application or when the definition file is introduced together with the application, the smartphone 1 analyzes the definition file when the application is introduced. And the smart phone 1 preserve | saves the information regarding the gesture obtained by analysis in the operation guide information 9Y, and refers to the information, when displaying a screen based on a definition file. The smartphone 1 may be configured to analyze the definition file and acquire information regarding the gesture every time the screen is displayed based on the definition file.

  When the definition file is downloaded from another device as needed, the smartphone 1 analyzes the definition file when the definition file is downloaded or when the screen is displayed based on the definition file, and acquires information on the gesture To do.

  In the example illustrated in FIG. 12, the name of the handler and a character string representing an outline of the processing included in the handler are set in association with the attribute of the tag related to the gesture. However, in the tag attribute related to the gesture, either the name of the handler or a character string representing an outline of the process included in the handler may be set.

  The operation guide information 9Y may be created based on the operation history as shown in FIG. In the example illustrated in FIG. 13, the smartphones 2 a to 2 n are portable electronic devices having the same configuration as the smartphone 1. When the gestures on the touch screen are detected, the smartphones 2a to 2n transmit information regarding the detected gestures to the server 80. Information transmitted to the server 80 includes screen information, operation information, and area information.

  The screen information is information for identifying the screen displayed on the display when the gesture is detected. The operation information is information for identifying the detected gesture. The area information is information indicating the display area of the element on the screen displayed at the position where the gesture is detected.

  The server 80 accumulates information transmitted from the smartphones 2 a to 2 n as operation history information 82 in the database 81. Then, based on the operation history information 82, the server 80 calculates a statistical value related to the number of times each gesture has been executed for each area of the screen. The statistical value is, for example, the total number of times, the frequency, the average value of the number of times for each smartphone, or the maximum value of the number of times for each smartphone. The server 80 determines that the large gesture having the calculated statistical value higher than the threshold is a gesture assigned to the corresponding screen area. The server 80 transmits the combination of the screen, the area on the screen, and the gesture determined to be assigned to the area to the portable electronic device including the smartphone 1 as the operation guide information 9Y.

  As described above, by generating the operation guide information 9Y based on the operation histories of a plurality of users, it is possible to share the knowledge regarding the gesture allocation noticed by each user with many users.

  In the example illustrated in FIG. 13, the portable electronic device that transmits information to the server 80 and the portable electronic device that receives information from the server 80 are illustrated as separate portable electronic devices, but these are the same portable electronic device. It may be. That is, the server 80 may also receive information about the detected gesture from the smartphone 1. The server 80 may transmit the operation guide information 9Y to the smartphones 2a to 2n.

  Next, an example of a processing procedure for displaying an operation guide will be described with reference to FIG. FIG. 14 is a flowchart illustrating an example of a processing procedure for displaying an operation guide. The processing procedure shown in FIG. 14 is realized by the controller 10 executing the control program 9A. In the processing procedure shown in FIG. 14, the operation guide is displayed when the screen is displayed, and is erased after a set time has elapsed. The controller 10 executes another processing procedure in parallel with the processing procedure shown in FIG. The other processing procedures include, for example, a processing procedure for displaying a screen and a processing procedure for executing processing corresponding to a gesture on the screen.

  The controller 10 determines whether the screen displayed on the display 2A was changed as step S101. If the screen has not been changed (No at Step S101), the controller 10 proceeds to Step S108.

  When the screen is changed (step S101, Yes), the controller 10 specifies the changed screen as step S102. And controller 10 judges whether it is necessary to display an operation guide about the specified screen as Step S103. For example, the controller 10 determines that it is not necessary to display the operation guide when the number of times the operation guide is displayed on the screen is greater than the preset number. When it is not necessary to display the operation guide (No at Step S104), the controller 10 proceeds to Step S108.

  When the operation guide needs to be displayed (step S104, Yes), the controller 10 displays the operation guide corresponding to the displayed screen as step S105. Then, the controller 10 determines whether a preset time has elapsed since the operation guide was displayed. When the preset time has not elapsed (step S106, No), the controller 10 repeats the determination of step S106.

  When the preset time has elapsed (step S106, Yes), the controller 10 deletes the operation guide as step S107. Then, the controller 10 returns to step S101.

  In step S108, the controller 10 determines whether or not an end instruction has been issued. The end instruction may be given by the user or may be made by the smartphone 1 itself. When the end instruction is given (step S108, Yes), the controller 10 ends the processing procedure. When the termination instruction has not been given (No at Step S108), the controller 10 returns to Step S101.

  Embodiment which this application discloses can be changed in the range which does not deviate from the summary and range of invention. Furthermore, the embodiment disclosed in the present application and its modifications can be combined as appropriate. For example, the above embodiment may be modified as follows.

  For example, each program shown in FIG. 5 may be divided into a plurality of modules, or may be combined with other programs.

  When displaying information about a gesture for each area to which a gesture is assigned, the smartphone 1 displays information about the gesture for an area whose area is narrower than the set value even if the gesture is assigned to that area. It does not have to be. For example, in the case of the example illustrated in FIG. 7, the smartphone 1 may not display the balloons 63d to 63f corresponding to the icons 63a to 63c whose display areas are narrow. By configuring in this way, it is possible to prevent the screen from being occupied by information related to the gesture and becoming difficult to see the screen.

  When the number of gestures assigned to the screen increases due to an application update, the smartphone 1 may display only information related to the increased gesture when displaying the screen. Alternatively, in this case, when the screen is displayed, the information regarding the increased gesture may be displayed in a mode different from the information regarding other gestures. Information regarding the increased gesture can be acquired by comparing the operation guide information 9Y before the application update and the operation guide information 9Y after the application update.

  The smartphone 1 may store the number of times detected for each gesture assigned to the screen. And the smart phone 1 may control the display of the information regarding a gesture based on the frequency | count memorize | stored. For example, when the corresponding screen is displayed, the smartphone 1 may display only information related to gestures in which the detected number of times is less than the set number of times. In this case, when the number of times the screen is displayed exceeds a threshold value, the smartphone 1 may be configured not to display the operation guide even if the detected number is smaller than the set number.

  In the above embodiment, a smartphone has been described as an example of a device including a touch screen, but the device according to the appended claims is not limited to a smartphone. The device according to the appended claims may be a portable electronic device other than a smartphone. Examples of portable electronic devices include, but are not limited to, mobile phones, tablets, portable personal computers, digital cameras, media players, electronic book readers, navigators, and game machines. The device according to the appended claims may be a stationary electronic device. The stationary electronic device includes, for example, a desktop personal computer, an automatic depositing and paying machine (ATM), and a television receiver, but is not limited thereto.

  The characterizing embodiments have been described in order to fully and clearly disclose the technology according to the appended claims. However, the appended claims should not be limited to the above-described embodiments, but all modifications and alternatives that can be created by those skilled in the art within the scope of the basic matters shown in this specification. Should be configured to embody such a configuration.

DESCRIPTION OF SYMBOLS 1 Smart phone 2 Touch screen display 2A Display 2B Touch screen 3 Button 4 Illuminance sensor 5 Proximity sensor 6 Communication unit 7 Receiver 8 Microphone 9 Storage 9A Control program 9B Mail application 9C Browser application 9D SNS application 9Z Setting data 10 Controller 11 Speaker 12, 13 Camera 14 Connector 15 Acceleration sensor 16 Direction sensor 17 Gyroscope 20 Housing 40 Home screen 50 Icon

Claims (7)

A display that displays a screen with user gestures assigned to some areas;
A touch screen for detecting the gesture;
A controller that executes processing corresponding to the gesture when the gesture is detected in the partial area;
The controller is configured to display the gesture assigned to each of the plurality of partial areas displayed on the screen when the partial area to which the gesture is assigned is displayed on the display. In association with each of the plurality of partial areas displayed on the screen, further display on the display ,
The controller displays the new gesture when the displayed screen is changed and a new gesture is assigned to the screen, and the number of times that the new gesture is displayed is a preset number of times. A device that does not display the new gesture when the number is exceeded .   The apparatus according to claim 1, wherein the controller erases the gesture when a preset time has elapsed.   The apparatus according to claim 1, wherein the controller deletes the gesture when a first operation is detected. Wherein the controller, when the second operation is detected, apparatus according to any one of claims 1 to 3 for displaying the gesture. The apparatus according to any one of claims 1 to 4 , wherein the controller displays the gesture when an operation is not detected for a longer period than a predetermined period. A method for controlling a device having a display and a touch screen, comprising:
Displaying on the display a screen in which a gesture by a user is assigned to some area;
Detecting the gesture by the touch screen;
When a plurality of the partial areas to which the gesture is assigned are displayed on the display, the gestures assigned to each of the partial areas displayed on the screen are displayed on the screen. Further displaying on the display in association with each of the plurality of the partial regions,
Executing the process corresponding to the gesture when the gesture is detected when the screen is displayed ;
When the screen displayed on the display is changed and a new gesture is assigned to the screen, the number of times that the new gesture is displayed on the display and the new gesture is displayed is preset. And not displaying the new gesture on the display when the number of times is exceeded . In a device having a display and a touch screen,
Displaying on the display a screen in which a gesture by a user is assigned to some area;
Detecting the gesture by the touch screen;
When a plurality of the partial areas to which the gesture is assigned are displayed on the display, the gestures assigned to each of the partial areas displayed on the screen are displayed on the screen. Further displaying on the display in association with each of the plurality of the partial regions,
Executing the process corresponding to the gesture when the gesture is detected when the screen is displayed ;
When the screen displayed on the display is changed and a new gesture is assigned to the screen, the number of times that the new gesture is displayed on the display and the new gesture is displayed is preset. A program that executes a step of not displaying the new gesture on the display when the number of times exceeds the predetermined number .

Images