JP6471261B1 - Electronic device, control method and program - Google Patents

Abstract

An electronic apparatus, a control method, and a program that improve operability for an input operation.
An electronic device 1 includes a sensor (proximity sensor 18) that detects a gesture without touching the device itself, a touch sensor 26, a display 14 that displays a first screen and a second screen, and a proximity sensor 18. And a controller 11 that associates the gesture detected by the proximity sensor 18 and the touch detected by the touch sensor with the operation on the first screen and the operation on the second screen, respectively, according to the arranged position.
[Selection] Figure 1

Description

  The present disclosure relates to an electronic device, a control method, and a program.

  For example, electronic devices such as smartphones and tablet terminals generally include a touch panel. A user generally controls such an electronic device by touching a touch panel. 2. Description of the Related Art In recent years, there has been known an electronic device that detects a gesture performed by a user away from an electronic device using a proximity sensor such as an infrared sensor and performs an input operation using the gesture (for example, Patent Document 1).

Japanese Patent Laying-Open No. 2015-225493

  Such an electronic device can perform, for example, an operation of scrolling the screen by touching the touch panel or by a gesture. For example, when the hand is dirty, the user can scroll the screen without touching the touch panel by enabling the input operation by the gesture.

  Here, in recent years, electronic devices that display a plurality of screens on a touch panel display have been used. In such an electronic device, for example, more operations such as operation of each screen, screen switching, screen position change, and the like can be executed. In order to improve the operability of such an electronic device, various inputs corresponding to many operations are required.

  Further, when the electronic device has a function of displaying information, the user may want to customize the scroll amount, for example. In general, it is possible to make settings on a menu screen provided in an electronic device, but it takes time and effort for the user to select hierarchical items and the like, and it does not support a case where a plurality of scroll amounts are desired to be used together. In order to improve the operability of such an electronic device, various inputs corresponding to different movement amounts are required.

  An object of the present disclosure made in view of such circumstances is to provide an electronic device, a control method, and a program in which operability is improved with respect to an input operation.

  The electronic device according to the present disclosure includes a sensor that detects a gesture without touching the device itself, a touch sensor, a display that displays the first screen and the second screen, and the sensor according to a position where the sensor is disposed. And a controller that associates the gesture detected by the touch sensor and the touch detected by the touch sensor with the operation on the first screen and the operation on the second screen, respectively.

  The control method of the present disclosure is a control method for an electronic device including a sensor that detects a gesture without touching the device itself, a touch sensor, and a display that displays a first screen and a second screen. And a step of associating a gesture detected by the sensor and a touch detected by the touch sensor with the operation on the first screen and the operation on the second screen, respectively, according to the position where is arranged.

  A program according to the present disclosure is provided in an electronic device that includes a sensor that detects a gesture without touching the device itself, a touch sensor, and a display that displays a first screen and a second screen. In response, the step of associating the gesture detected by the sensor and the touch detected by the touch sensor with the operation on the first screen and the operation on the second screen, respectively, is executed.

  According to an embodiment of the present disclosure, it is possible to provide an electronic device, a control method, and a program that have improved operability for an input operation.

It is a schematic block diagram of the electronic device which concerns on one Embodiment. It is a figure which illustrates a mode that a user operates an electronic device with gesture. It is a schematic block diagram of a proximity sensor. It is a figure which shows transition of the detected value which each infrared photodiode detects. It is a figure which illustrates the condition of operating an electronic device with a gesture. It is a figure which illustrates a mode that a user operates an electronic device with gesture. It is a figure which illustrates the condition of operating an electronic device with a gesture. It is a figure which illustrates the screen display of an electronic device. It is a figure which shows an example of operation by gesture. It is a figure which shows an example of operation by a touch. It is a figure which shows an example of operation by gesture. It is a figure which shows an example of operation by a touch. It is a figure which shows an example of operation by the gesture after operation of FIG. It is a figure which shows an example of operation by a touch. It is a figure which shows an example of operation by a touch. It is a figure which shows an example of operation by gesture. It is a figure which shows an example of operation by a touch. It is a figure which shows an example of operation by gesture. It is a figure which shows an example of operation by gesture and a touch. It is a flowchart which illustrates the process according to the user operation | movement of an electronic device. It is a figure for demonstrating the difference in the moving amount | distance of a screen. It is a figure which shows an example of operation by a touch. It is a figure which shows an example of operation by gesture. It is a figure which shows an example of operation by a touch. It is a figure which shows an example of operation by gesture. It is a figure which shows an example of operation by gesture. It is a flowchart which illustrates the process according to the user operation | movement of an electronic device. It is a flowchart which illustrates the process according to the user operation | movement of an electronic device. It is a figure which shows a ranging sensor typically. It is a figure which shows typically an example of arrangement | positioning of the light receiving element in the light-receiving part shown in FIG. It is a figure which shows typically transition of the distance to the target object detected by each light receiving element.

(Configuration of electronic equipment)
As shown in FIG. 1, the electronic apparatus 1 according to an embodiment includes a proximity sensor 18 (gesture sensor), a touch sensor 26, and a controller 11. The electronic device 1 includes a timer 12, a camera 13, a display 14, a microphone 15, a storage 16, a communication unit 17, and a speaker 25. The electronic device 1 further includes a UV sensor 19, an illuminance sensor 20, an acceleration sensor 21, a geomagnetic sensor 22, an atmospheric pressure sensor 23, and a gyro sensor 24. FIG. 1 is an illustration. The electronic device 1 may not include all of the components shown in FIG. Further, the electronic device 1 may include components other than those shown in FIG.

  The timer 12 receives a timer operation instruction from the controller 11 and outputs a signal indicating that to the controller 11 when a predetermined time has elapsed. As shown in FIG. 1, the timer 12 may be provided independently of the controller 11 or may be configured to be built in the controller 11.

  The camera 13 images a subject around the electronic device 1. As an example, the camera 13 is an in-camera provided on a surface on which the display 14 of the electronic device 1 is provided. Moreover, the camera 13 is an out camera provided on the back surface (surface opposite to the surface on which the display 14 is provided) of the casing of the electronic device 1 as an example. The camera 13 may include an in camera and an out camera.

  The display 14 displays a screen. The screen includes at least one of, for example, a character, an image, a symbol, and a graphic. The display 14 may be a liquid crystal display. The display 14 may be an organic EL panel (Organic Electro-Luminescence Panel) or an inorganic EL panel (Inorganic Electro-Luminescence Panel). In the present embodiment, the display 14 is a touch panel display (touch screen display) integrated with the touch sensor 26.

  The microphone 15 detects sounds around the electronic device 1 including a voice uttered by a person.

  The storage 16 stores programs and data as a storage unit. The storage 16 temporarily stores the processing result of the controller 11. The storage 16 may include any storage device such as a semiconductor storage device and a magnetic storage device. The storage 16 may include a plurality of types of storage devices. The storage 16 may include a combination of a portable storage medium such as a memory card and a storage medium reading device.

  The programs stored in the storage 16 include an application executed in the foreground or the background and a control program that supports the operation of the application. For example, the application causes the controller 11 to execute processing according to the gesture. The control program is, for example, an OS (Operating System). The application and control program may be installed in the storage 16 via communication by the communication unit 17 or via a storage medium. The data stored in the storage 16 includes various information such as information such as still images (photos), moving images, maps, highway ICs (Inter Change), and traffic information. For example, the map and traffic information can be obtained and updated via the communication unit 17.

  The communication unit 17 is an interface for communicating by wire or wireless. The communication method performed by the communication unit 17 of one embodiment is a wireless communication standard. For example, the wireless communication standards include cellular phone communication standards such as 2G, 3G, and 4G. For example, cellular phone communication standards include LTE (Long Term Evolution) and W-CDMA (Wideband Code Division Multiple Access). Further, for example, cellular phone communication standards include CDMA2000 and PDC (Personal Digital Cellular). Further, for example, cellular phone communication standards include GSM (registered trademark) (Global System for Mobile communications) and PHS (Personal Handy-phone System). For example, wireless communication standards include WiMAX (Worldwide Interoperability for Microwave Access), IEEE 802.11, and Bluetooth (registered trademark). Further, for example, wireless communication standards include IrDA (Infrared Data Association), NFC (Near Field Communication), and the like. The communication unit 17 can support one or more of the communication standards described above.

  The speaker 25 outputs sound. When the electronic device 1 is a device capable of making a call, for example, the voice of the other party is output from the speaker 25 during a call. For example, when reading out news or weather forecasts, the content is output from the speaker 25 as sound.

  The proximity sensor 18 detects the relative distance to the objects around the electronic device 1 and the moving direction of the objects in a non-contact manner. In the present embodiment, the proximity sensor 18 includes one light source infrared LED (Light Emitting Diode) and four infrared photodiodes. The proximity sensor 18 irradiates the target with infrared light from the light source infrared LED. The proximity sensor 18 uses reflected light from the object as incident light of an infrared photodiode. The proximity sensor 18 can measure the relative distance to the object based on the output current of the infrared photodiode. Further, the proximity sensor 18 detects the moving direction of the object based on a time difference in which the reflected light from the object enters each infrared photodiode. Therefore, the proximity sensor 18 can detect an operation using an air gesture (hereinafter simply referred to as “gesture”) performed by the user of the electronic device 1 without touching the electronic device 1. Here, the proximity sensor 18 may include a visible light photodiode.

  The touch sensor 26 is a sensor that detects contact of a user's finger or stylus pen and specifies the contact position. As described above, in the present embodiment, the touch sensor 26 is integrated with the display 14 to constitute a touch panel display. The touch sensor 26 can simultaneously detect a plurality of positions touched by a finger or a stylus pen.

  The controller 11 is a processor such as a CPU (Central Processing Unit). The controller 11 may be an integrated circuit such as an SoC (System-on-a-Chip) in which other components are integrated. The controller 11 may be configured by combining a plurality of integrated circuits. The controller 11 controls various operations of the electronic device 1 to realize various functions. Here, when the electronic device 1 is a device constituting a car navigation system mounted on a vehicle, the controller 11 is a CPU and constitutes an ECU (Electronic Control Unit) together with the communication unit 17 and the storage 16. Can do.

  Specifically, the controller 11 refers to data stored in the storage 16 as necessary. The controller 11 implements various functions by executing instructions included in the program stored in the storage 16 and controlling other functional units such as the display 14. For example, the controller 11 acquires contact data from the user from the touch sensor 26. For example, the controller 11 acquires information related to the user's gesture detected by the proximity sensor 18. For example, the controller 11 grasps the activation status of the application.

  The UV sensor 19 can measure the amount of ultraviolet rays contained in sunlight or the like.

  The illuminance sensor 20 detects the illuminance of ambient light incident on the illuminance sensor 20. The illuminance sensor 20 may use a photodiode, for example. Further, the illuminance sensor 20 may be one using a phototransistor, for example.

  The acceleration sensor 21 detects the direction and magnitude of acceleration acting on the electronic device 1. The acceleration sensor 21 uses the detected acceleration information as an output signal. The acceleration sensor 21 is, for example, a three-axis (three-dimensional) type that detects acceleration in the x-axis direction, the y-axis direction, and the z-axis direction. The acceleration sensor 21 may be a piezoresistive type, for example. Further, the acceleration sensor 21 may be, for example, a capacitance type.

  The geomagnetic sensor 22 detects the direction of geomagnetism and makes it possible to measure the direction of the electronic device 1.

  The atmospheric pressure sensor 23 detects the atmospheric pressure (atmospheric pressure) outside the electronic device 1.

  The gyro sensor 24 detects the angular velocity of the electronic device 1. The controller 11 can measure the change in the orientation of the electronic device 1 by time-integrating the angular velocity acquired by the gyro sensor 24.

(Operating electronic devices using gestures)
FIG. 2 shows a state in which the user operates the electronic device 1 with a gesture. In FIG. 2, the electronic device 1 is supported by a stand as an example. As an alternative, the electronic device 1 may be leaned against a wall or placed on a table. When the proximity sensor 18 detects a user gesture, the controller 11 performs processing based on the detected gesture. In the example of FIG. 2, the process based on the gesture is scrolling of the screen on which the recipe is displayed. For example, when the user performs a gesture of moving his / her hand upward in the longitudinal direction of the electronic device 1, the screen scrolls upward in conjunction with the movement of the user's hand. Further, for example, when the user performs a gesture of moving the hand downward in the longitudinal direction of the electronic device 1, the screen scrolls downward in conjunction with the movement of the user's hand. Here, instead of the gesture detection by the proximity sensor 18, the controller 11 may execute the gesture detection based on an image captured by the camera 13.

  In the example of FIG. 2, the screen is divided into two. In the example of FIG. 2, the recipe is displayed on the upper screen of the two screens, and can be scrolled by the gesture as described above. On the lower screen of the two screens, a home screen on which a plurality of icons are arranged is displayed. The user can start an application associated with the icon by touching the icon on the lower screen. In this specification, a user's touch operation (contact operation) on the touch panel display is simply expressed as touch. The touch includes various operations such as a tap, a double tap, a long tap (long press), and a slide.

  The electronic device 1 shown in FIG. 2 is a smartphone. As an alternative example, the electronic device 1 may be, for example, a mobile phone terminal, a Fablet, a tablet PC, or a feature phone. The electronic device 1 is not limited to the above, and may be, for example, a PDA, a remote control terminal, a portable music player, a game machine, an electronic book reader, a car navigation, a home appliance, or an industrial device (FA device).

(Gesture detection method by proximity sensor)
Here, a method in which the controller 11 detects a user gesture based on the output of the proximity sensor 18 will be described with reference to FIGS. 3 and 4. FIG. 3 is a diagram illustrating a configuration example of the proximity sensor 18 when the electronic apparatus 1 is viewed from the front. The proximity sensor 18 includes a light source infrared LED 180 and four infrared photodiodes SU, SR, SD, and SL. The four infrared photodiodes SU, SR, SD, and SL detect reflected light from the detection target via the lens 181. The four infrared photodiodes SU, SR, SD, and SL are arranged symmetrically when viewed from the center of the lens 181. Here, it is assumed that the virtual line D <b> 1 shown in FIG. 3 is substantially parallel to the longitudinal direction of the electronic device 1. On the virtual line D1 in FIG. 3, the infrared photodiode SU and the infrared photodiode SD are arranged apart from each other. In the direction of the imaginary line D1 in FIG. 3, the infrared photodiodes SR and SL are arranged between the infrared photodiode SU and the infrared photodiode SD.

  FIG. 4 shows the transition of the detection value when the detection objects (for example, the user's hand) of the four infrared photodiodes SU, SR, SD, and SL move along the direction of the virtual line D1 in FIG. Illustrate. Here, the infrared photodiode SU is farthest from the infrared photodiode SD in the direction of the virtual line D1. Therefore, as shown in FIG. 4, the time difference between the change (for example, increase) in the detection value (broken line) of the infrared photodiode SU and the same change (for example, increase) in the detection value (thin solid line) of the infrared photodiode SD. Is the largest. The controller 11 can determine the moving direction of the detection object by grasping the time difference of the predetermined change in the detection values of the photodiodes SU, SR, SD, and SL.

  The controller 11 acquires the detection values of the photodiodes SU, SR, SD, and SL from the proximity sensor 18. Then, the controller 11 integrates, for a predetermined time, a value obtained by subtracting the detection value of the photodiode SU from the detection value of the photodiode SD in order to grasp the movement of the detection object in the direction of the virtual line D1, for example. Good. In the example of FIG. 4, the integral value is a non-zero value in the regions R41 and R42. From the change in the integrated value (for example, change in positive value, zero, and negative value), the controller 11 can grasp the movement of the detection target in the direction of the virtual line D1.

  Further, the controller 11 may integrate a value obtained by subtracting the detection value of the photodiode SR from the detection value of the photodiode SL in a predetermined time. From this change in integral value (for example, a change in positive value, zero, or negative value), the controller 11 moves the detection target in a direction orthogonal to the virtual line D1 (a direction substantially parallel to the short direction of the electronic device 1). Can be grasped.

  As an alternative example, the controller 11 may perform calculation using all the detected values of the photodiodes SU, SR, SD, and SL. In other words, the controller 11 may grasp the moving direction of the detection object without calculating it by separating the components in the longitudinal direction and the short direction of the electronic device 1.

  The gestures detected by the proximity sensor 18 include, for example, left and right gestures, up and down gestures, diagonal gestures, gestures that draw a circle in a clockwise direction, and gestures that draw a circle in a counterclockwise direction. For example, the right and left gesture is a gesture performed in a direction substantially parallel to the short direction of the electronic device 1. The upper and lower gestures are gestures performed in a direction substantially parallel to the longitudinal direction of the electronic device 1. The oblique gesture is a gesture performed in a direction that is not parallel to either the longitudinal direction or the short direction of the electronic device 1 on a plane substantially parallel to the electronic device 1.

(Kitchen mode)
FIG. 5 shows an example of a situation where the user operates the electronic device 1 with a gesture. In the example of FIG. 5, the user is cooking according to the recipe in the kitchen while displaying the cooking recipe on the display 14 of the electronic device 1. In the example of FIG. 5, the proximity sensor 18 detects a user's gesture. Then, the controller 11 performs processing based on the gesture detected by the proximity sensor 18. For example, the controller 11 can perform a process of scrolling the recipe according to a specific gesture (for example, a gesture in which the user moves the hand up and down). During cooking, the user's hand may become dirty or wet. However, the user can scroll the recipe without touching the electronic device 1. Therefore, it is possible to avoid the display 14 from becoming dirty and the stain of the display 14 from being transferred to the user's hand during cooking.

  Here, the electronic device 1 has a plurality of modes. The mode means an operation mode (an operation state or an operation state) that restricts the overall operation of the electronic device 1. Only one mode can be selected at a time. In the present embodiment, the modes of the electronic device 1 include at least a first mode, a second mode, and a third mode. The first mode is a normal operation mode (normal mode). The second mode is an operation mode (kitchen mode) of the electronic device 1 that is optimal for cooking while displaying a recipe in the kitchen. In addition, the third mode is an operation mode (car mode) of the electronic device 1 that is provided in a moving body, particularly a vehicle, and is optimal for providing various information (for example, information useful for driving or maneuvering a map or the like). . As described above, in the case of the second mode, an operation by a gesture is possible. As will be described below, gesture operation is also possible in the third mode. Here, the mobile body may include, for example, a vehicle, a ship, an aircraft, and the like. Vehicles can include, for example, electric vehicles, hybrid electric vehicles, gasoline vehicles, motorcycles, motorcycles, welfare vehicles, and the like. The vehicle may include a rail vehicle, for example. The mobile may be driven or steered by the user. At least a part of user operations related to driving or maneuvering of the mobile object may be automated.

(Car mode)
FIG. 6 shows a state in which the user operates the electronic device 1 operating in the car mode with a gesture. In FIG. 6, the electronic device 1 is installed on a console panel of an automobile as an example. As an alternative, the electronic device 1 may be indicated by a support provided in the automobile. The support may be placed on the dashboard. Moreover, as shown in FIG. 6, the electronic device 1 that operates in the car mode may be used in a horizontal direction (a direction in which the longitudinal direction is left and right). When the proximity sensor 18 detects a user gesture, the controller 11 performs processing based on the detected gesture. In the example of FIG. 6, the process based on the gesture is movement of a screen on which a wide area map is displayed. For example, when the user performs a gesture of moving the hand to the left in the longitudinal direction of the electronic device 1, the screen moves to the left in conjunction with the movement of the user's hand. Here, the correspondence between the direction of the gesture for moving the user's hand and the direction in which the screen moves can be arbitrarily set. For example, when the user performs a gesture of moving his hand to the left, the screen may move to the right instead of the left.

  As described above, the electronic device 1 illustrated in FIG. 6 is a smartphone that can be installed and removed from the console panel. As another example, the electronic device 1 may be a car navigation device installed in a vehicle. At this time, the mode of the electronic device 1 may not include the second mode.

  FIG. 7 shows an example of a situation in which the user operates the electronic device 1 with a gesture. As shown in FIG. 7, for example, the electronic device 1 is arranged so that the display 14 is located at the center of the console panel of the automobile. In the example of FIG. 7, in a state where the route to the destination is displayed on the display 14 of the electronic device 1, the user gets into the automobile on which the electronic device 1 is mounted while referring to the displayed route, Drive a car. At this time, the proximity sensor 18 is in a state in which the user's gesture can be detected. The controller 11 performs processing based on the gesture detected by the proximity sensor 18.

  The electronic device 1 can accept a touch from a user on the touch panel display. However, it is not preferable that the user shifts his / her line of sight to the display 14 during driving in order to confirm the touch position. The electronic device 1 according to the present disclosure enables a user to perform an operation using a gesture. Therefore, the user during driving does not look at the display 14 for a long time.

  Here, when the electronic device 1 operates in the kitchen mode or the car mode, the user can operate the electronic device 1 using a gesture and a touch. The electronic device 1 according to the present disclosure relates to an input operation when, for example, displaying a plurality of screens or adjusting a moving amount of a screen by associating a gesture and a touch with a screen operation as described below. Operability can be improved.

(First method)
FIG. 8 shows an example of a screen display of the electronic device 1. As shown in FIG. 8, the display 14 of the electronic device 1 displays a first screen 141 and a second screen 142. That is, the electronic device 1 has a multi-window function, and displays a screen divided on the display 14. Each of the first screen 141 and the second screen 142 can execute an application independently of each other. In the example of FIG. 8, a recipe for cooking is displayed on the first screen 141. In the example of FIG. 8, a home screen on which a plurality of icons 30 are arranged is displayed on the second screen 142. Moreover, the electronic device 1 is operating in the kitchen mode. Here, the first screen 141 and the second screen 142 may be partially overlapped and displayed. In the first method and the second method described later, the gesture detected by the proximity sensor 18 and the touch detected by the touch sensor 26 are associated with the operation on the first screen 141 and the operation on the second screen 142, respectively. Further, the touch sensor 26 is integrated with the display 14 as described above to constitute a touch panel display.

  As a first technique, the controller 11 of the electronic device 1 associates the gesture with the operation on the first screen 141. The controller 11 of the electronic device 1 associates the touch with the operation on the second screen 142. In the example of FIG. 8, the proximity sensor 18 is disposed on the upper part of the display 14 of the electronic apparatus 1, that is, on the side closer to the first screen 141 than the second screen 142. The controller 11 associates the touch with the operation on the second screen 142 by associating the gesture with the operation on the first screen 141 closer to the second screen 142 from the position where the proximity sensor 18 is disposed.

  Here, the controller 11 may store the gesture and the operation on the first screen 141 and the touch and the operation on the second screen 142 in the storage 16 in association with each other by, for example, a table. For example, the controller 11 causes the storage 16 to store in advance a table in which a gesture for moving the hand in the vertical direction and screen scrolling are associated with each other. In addition, the controller 11 stores in advance in the storage 16 a table in which double taps and application executions are associated. Then, the controller 11 reads such a table from the storage 16 and executes an operation associated with the gesture and the touch according to the table.

  FIG. 9 is a diagram illustrating an example of an operation by a gesture. In the first method, the controller 11 scrolls the cooking recipe displayed on the first screen 141 in the vertical direction when the proximity sensor 18 detects a gesture for moving the hand in the vertical direction. The scroll direction of the first screen 141 according to the direction in which the gesture hand is moved (for example, upward) may be the same (for example, upward) or the reverse direction (for example, downward).

  Here, the first screen 141 may be displayed in a different manner from the second screen 142 so that the user can distinguish the screen (first screen 141) that can be operated with the gesture. In the present embodiment, the first screen 141 is different in frame color from the second screen 142. For example, the frame color of the first screen 141 is red. For example, the frame of the second screen 142 is black. In the drawings after FIG. 9, the double frame of the first screen 141 means that the color of the frame is different. Here, as another example, a specific pictograph or the like may be displayed on the first screen 141 (or the second screen 142) so that the user can distinguish.

  Further, the display mode of the frame of the first screen 141 may be changed to indicate to the user that a gesture has been detected by the proximity sensor 18. For example, when the gesture is detected, the frame of the first screen 141 has a line thickness (for example, becomes thicker), a type (for example, becomes a dotted line, blinks), or a color (for example, changes from red to blue). ) May change.

  FIG. 10 is a diagram illustrating an example of an operation by touch. In the first method, the controller 11 scrolls the home screen on which the icons 30 displayed on the second screen 142 are lined up and down, for example, when the touch sensor 26 detects a slide in the vertical direction.

(Second method)
In the first method, the controller 11 associates the gesture with the operation on the first screen 141 closer to the second screen 142 from the position where the proximity sensor 18 is disposed. Here, the first screen 141 may be farther than the second screen 142 from the position where the proximity sensor 18 is disposed. In the second method, the controller 11 associates the gesture with the operation on the second screen 142 by associating the gesture with the operation on the first screen 141 farther than the second screen 142 from the position where the proximity sensor 18 is disposed.

  FIG. 11 is a diagram illustrating an example of an operation by a gesture in the second method. In the second method, the controller 11 displays a home screen on which icons 30 displayed on the first screen 141 far from the proximity sensor 18 are arranged when a gesture of moving the hand in the vertical direction is detected by the proximity sensor 18. Scroll up and down. Here, the scroll direction of the first screen 141 corresponding to the direction in which the gesture hand is moved (for example, upward) may be the same (for example, upward) or the reverse direction (for example, downward). In the second method, the user can scroll the recipe of the dish displayed on the second screen 142 on the side close to the proximity sensor 18 by touch.

  In the first method and the second method, the controller 11 associates the gesture and the touch with the operation on the first screen 141 and the operation on the second screen 142 according to the position where the proximity sensor 18 is arranged. That is, each of the divided screens can be operated by a gesture or a touch. In particular, as shown in FIG. 8, when the display 14 displays two screens, one screen is operated by a gesture and the other screen is operated by a touch, so that the user can easily grasp the operation method. it can.

  Here, the user may be able to select the first method or the second method depending on the setting of the electronic device 1 that can be selected from the menu screen, for example. In the first method, the user operates the first screen 141 close to the position where the proximity sensor 18 is arranged with a gesture. Since the user moves his hand near the proximity sensor 18 (executes a gesture), the screen to be operated (first screen 141) is easy to understand intuitively. In the second method, the user operates the first screen 141 far from the position where the proximity sensor 18 is arranged with a gesture. The user can easily view the screen (the first screen 141) because the user does not block the first screen 141, which is the operation target of the gesture, with his / her hand (not covered with the hand performing the gesture). The user may select the first method when he likes that the screen to be operated is intuitively easy to understand. In addition, the user may select the second method when he prefers that the screen is easy to see.

(Third method)
In the third method, the controller 11 associates the gesture with the operation on the first screen 141 closer to the second screen 142 from the position where the proximity sensor 18 is arranged, and the touch is performed not only on the operation on the second screen 142. Further, it is associated with at least one of an operation between screens and a screen erasing operation. This is the same as the first method except that at least one of operations between screens and screen erasing operations is executed by touch. Here, the operation between screens is an operation executed between a plurality of screens. The operation between the screens includes, for example, replacement of display contents described later, copying of display contents, movement of display contents, and the like.

  As described above, touch includes various operations such as a tap, a double tap, a long tap, and a slide. Further, as described above, the controller 11 associates a part of the touch with the operation of the second screen 142. In the third method, the controller 11 associates a touch that is not associated with an operation on the second screen 142 with at least one of an operation between screens and a screen erasing operation. For example, the controller 11 may associate the slide in the vertical direction after the long tap with the replacement of the display contents of the first screen 141 and the second screen 142. For example, the controller 11 may associate the slide in the left-right direction after the long tap with the deletion of the second screen 142. The deletion of the second screen 142 ends the application displayed on the second screen 142.

  FIG. 12 is a diagram illustrating an example of an operation by touch in the third technique. In the third method, the controller 11 changes the display so that the second screen 142 overlaps the first screen 141 when the user slides upward after a long tap on the second screen 142. When the user further slides the second screen 142 upward, the display contents of the first screen 141 and the second screen 142 are switched. That is, a home screen in which a plurality of icons 30 are arranged on the first screen 141 is displayed. In addition, a recipe for cooking is displayed on the second screen 142.

  FIG. 13 is a diagram illustrating a case where an operation by a gesture is executed after the operation between the screens in FIG. 12. When the gesture that moves the hand in the up and down direction is detected by the proximity sensor 18, the controller 11 scrolls the home screen displayed on the first screen 141 in the up and down direction.

  Moreover, FIG. 14 is a figure which shows another example of operation by the touch in a 3rd method. In the third method, the controller 11 changes the display so that the second screen 142 moves to the right when the user slides rightward after long-tapping the second screen 142. When the user further slides the second screen 142 to the right, the second screen 142 is not displayed on the display 14 and is erased.

  Here, in the third method, as another example, the controller 11 may associate the gesture with the operation on the first screen 141 farther than the second screen 142 from the position where the proximity sensor 18 is disposed. At this time, it is the same as the second method except that at least one of operations between screens and screen erasing operations is executed by touch.

  As yet another example, the controller 11 may associate a touch with at least one of an operation between screens and a screen erasing operation without associating the touch with an operation on the second screen 142.

  As yet another example, the controller 11 may associate a gesture instead of a touch with at least one of an operation between screens and a screen erasing operation. For example, the controller 11 may associate the up / down direction gesture after the circle drawing gesture with the switching of the display contents of the first screen 141 and the second screen 142. For example, the controller 11 may associate the gesture in the left-right direction after the gesture for drawing a circle with the deletion of the second screen 142.

  In the third method, the controller 11 associates not only an operation on the first screen 141 or an operation on the second screen 142 but also an operation between screens and an operation for erasing the screen with a touch or a gesture. For this reason, the third method allows more various inputs.

(Fourth method)
In the fourth method, the controller 11 associates a specific touch that is one of the touches that can be executed on the second screen 142 with a specific operation on the second screen 142, and assigns one of the gestures to the first screen 141. Corresponds to specific operations. The specific operation is, for example, a notification display. One of the touches that can be executed on the second screen 142 is a slide from the outside of the second screen 142 to the inside of the second screen 142.

  FIG. 15 is a diagram illustrating an example of an operation by touch in the fourth technique. In the fourth method, the controller 11 displays a notification indicating notification when the user slides downward from the outside of the second screen 142 to the inside of the second screen 142. Here, in the example of FIG. 15, the first screen 141 is displayed at a position farther than the second screen 142 from the position where the proximity sensor 18 is arranged, as in the second method.

  Here, the fact that the notification is displayed so as to be pulled out by sliding from the outside to the inside of the screen enables an intuitive operation for the user. However, in the example of FIG. 15, when the user tries to slide down from the outside of the first screen 141 to the inside of the first screen 141, the user touches the second screen 142 above the first screen 141. Become. Then, the user's touch is treated as being for the second screen 142. That is, the notification cannot be displayed on the first screen 141 by the same touch as the second screen 142.

  Here, the electronic device 1 according to the present disclosure can be operated by a gesture. Therefore, when the specific touch on the second screen 142 cannot be applied to the first screen 141 as it is, the electronic device 1 can execute a specific operation on the first screen 141 by the gesture. In the above example, the specific touch is a downward slide from the outside of the screen to the inside of the screen. An operation associated with a specific touch (specific operation) is a notification display.

  FIG. 16 is a diagram illustrating an example of an operation by a gesture in the fourth technique. As shown in FIG. 16, the controller 11 associates one of the gestures for moving the hand downward with the display of the notification on the first screen 141.

  The fourth method is useful when a plurality of screens are displayed on the display 14 and a touch that is included in a region where the start point or the end point is different depending on the screen is used. The area is, for example, a display area on the screen. In the above example, when the notification is displayed on the second screen 142, a downward slide with the start point outside the display area of the screen (the first screen 141 and the second screen 142) is used. However, when a downward slide is executed on the first screen 141, the starting point is inside the display area of the screen. That is, it is included in the area where the starting point differs depending on the screen. In such a case, the controller 11 can execute a specific operation (for example, display of a notification) on any of a plurality of screens by using a gesture.

(Fifth technique)
In the fifth method, the controller 11 associates a touch and a gesture with the operation of the active window among the active window and the inactive window that are simultaneously displayed. Further, the controller 11 associates the gesture with the display of the list of setting items of the active window. Here, the active window is a screen in a state where the user can perform an operation. An inactive window is a screen in a state where the user cannot perform an operation. Here, an application or the like executed in the inactive window may operate in the background.

  FIG. 17 is a diagram illustrating an example of an operation by touch in the fifth technique. In the example of FIG. 17, the user touches the upper screen among a plurality of screens (upper screen and lower screen) displayed on the display 14. The upper screen touched is set as the active window. The lower screen is set as an inactive window. Here, when the user touches the lower screen, the lower screen is set as an active window. That is, the active window can be switched by touching a part of the screen.

  The active window corresponds to the first screen 141 and the second screen 142 described above. That is, the user can operate the active window by touch and gesture. The active window is displayed on the display 14 together with the third screen 143 that is an inactive window. As shown in FIG. 17, the controller 11 scrolls the displayed recipe of the dish in the vertical direction when a touch sliding in the vertical direction is detected by the touch sensor 26 in the active window.

  FIG. 18 is a diagram illustrating an example of an operation by a gesture in the fifth technique. As shown in FIG. 18, the controller 11 associates the gesture of moving the hand in the left-right direction (horizontal direction) with the display of a menu (list of setting items) in the active window. In the example of FIG. 18, the menu is displayed so as to be drawn rightward from the left end of the active window in response to the gesture of moving the hand to the right. Here, as another example, the menu may be displayed so as to be drawn leftward from the right edge of the active window in response to the gesture of moving the hand leftward. As another example, the notification may be displayed to be drawn downward from the upper end of the active window in response to the gesture of moving the hand downward.

  In the fifth method, an active window can be operated by touch and gesture. For this reason, the fifth method allows various inputs to one screen. In the fifth method, the controller 11 associates a gesture with at least a menu display. Therefore, when the user wants to change or confirm the setting item, the menu can be quickly displayed by the gesture.

  In the fifth method, as in the above example, the direct operation of the active window (for example, moving the screen such as scrolling or erasing the screen) is associated with the touch, and the auxiliary operation (for example, changing the menu or the like) (Display on the screen) may correspond to the gesture. Conversely, it is possible to make a direct operation correspond to a gesture and an auxiliary operation correspond to a touch. By associating with a touch or gesture according to the type of operation, the user can operate the active window more intuitively.

  Further, the direction or direction of movement may be different between the touch and the gesture. For example, while a touch is a movement in a first direction (for example, up and down direction), a gesture may use only a movement in a second direction (for example, left and right direction). Also, for example, while a touch is a movement in a first direction (downward and leftward as an example), a gesture may use only a movement in a second direction (upward and rightward as an example). Because touch and gestures can be used for one active window, the user may be confused if there is movement in the same direction or direction. Therefore, the user may be able to easily use the touch and the gesture easily by preventing the direction or direction from overlapping between the touch and the gesture.

(Sixth method)
Although FIG. 8 to FIG. 18 show the electronic device 1 when operating in the kitchen mode, the gesture and touch can be performed on the first screen 141 and the second screen even when the electronic device 1 operates in the car mode. 142 can be associated with each operation.

  FIG. 19 is a diagram illustrating an example of an operation by a gesture and touch in the sixth technique. In the example of FIG. 19, the proximity sensor 18 is on the driver's seat side and detects the driver's gesture well. On the first screen 141 closer to the proximity sensor 18, a map for guiding the navigation of the vehicle to the destination is displayed. The map shows the current position of the vehicle, the road name, and the like indicated by circles. An audio operation screen is displayed on the second screen 142 away from the proximity sensor 18. The audio operation screen shows, for example, the selected album, song name, volume button, and the like.

  As illustrated in FIG. 19, the controller 11 associates the gesture of moving the hand up, down, left, and right with the movement of the map on the first screen 141. In addition, the controller 11 associates the touch on the volume button (the button with the upward and downward arrows) with the adjustment of the volume on the second screen 142.

  The driver can move the map using gestures. The driver can operate the electronic device 1 without looking at the display 14 for a long time in order to confirm the touch position. In addition, on the second screen 142 on the passenger seat side, the user who does not drive can adjust the audio volume by touching.

  In the sixth method, the controller 11 associates the gesture and the touch with the operation on the first screen 141 and the operation on the second screen 142 according to the position of the proximity sensor 18 on the driver's seat side. At this time, the driver can safely operate the electronic device 1 using a gesture. The user on the passenger seat side can perform various operations by touching. Here, when the electronic apparatus 1 operates in the car mode, the position of the proximity sensor 18 is not limited to the position shown in FIG. For example, the proximity sensor 18 may be freely installed as long as it is closer to the driver seat side than the passenger seat side when the electronic device 1 is used.

(flowchart)
FIG. 20 is a flowchart illustrating a process corresponding to the user operation of the electronic device 1 corresponding to the first to sixth methods.

  When the first screen 141 and the second screen 142 are displayed on the display 14 (Yes in step S1), the controller 11 of the electronic device 1 proceeds to the process in step S2. Here, when the multi-window function is used by the user, the first screen 141 and the second screen 142 may be displayed. The controller 11 of the electronic device 1 also proceeds to the process of step S2 when an active window and an inactive window corresponding to the first screen 141 and the second screen 142 are displayed.

  When the first screen 141 and the second screen 142 are not displayed on the display 14 (No in step S1), the controller 11 of the electronic device 1 ends the process.

  When the proximity sensor 18 detects a gesture (Yes in step S2), the controller 11 of the electronic device 1 performs an operation on the first screen 141 associated with the gesture (step S3).

  When the proximity sensor 18 does not detect a gesture (No in step S2) or after the process in step S3, the controller 11 of the electronic device 1 proceeds to the process in step S4. Here, the controller 11 of the electronic device 1 may proceed to the process of step S4 when the gesture is not detected at a predetermined time (for example, 1 minute).

  When the touch sensor 26 detects a touch (Yes in step S4), the controller 11 of the electronic device 1 performs an operation on the second screen 142 associated with the touch (step S5).

  When the touch sensor 26 does not detect a touch (No in step S4) or after the process in step S5, the controller 11 of the electronic device 1 returns to the process in step S1. Here, the controller 11 of the electronic device 1 may return to the process of step S1 when the touch is not detected for a predetermined time (for example, 1 minute).

(Seventh method)
The controller 11 of the electronic device 1 can improve the operability of the electronic device 1 by making the gesture and the touch correspond to a plurality of different movement amounts. In the following, an embodiment in which a plurality of different movement amounts are used together in one application will be described. In the following, there may be a plurality of screens (multi-window), but a case where there is one screen will be described as an example. In addition, the electronic device 1 operates in a mode in which an operation by a gesture is possible.

  FIG. 21 shows an application screen for displaying a photograph. In the example of FIG. 21, thumbnails 31 obtained by reducing a plurality of photos are displayed in a list. The thumbnails 31 are numbered according to a predetermined order (for example, shooting date order). In the following, numbers are used to distinguish each of the photographs. For example, the first photo means a photo numbered “1” in FIG. In this application, for example, when the user taps the thumbnail 31 of the first photo, the first photo is displayed large on the display.

  In an application for displaying photos, when the number of photos increases, a user may need to make many touches before displaying a desired photo. For this reason, there has been a demand to increase the scroll amount of display (for example, how many sheets to skip). On the other hand, in such an application, the user needs a function of displaying the images one by one in order.

  In the seventh method, the controller 11 associates gestures and touches with operations that change the screen with different movement amounts. Specifically, the controller 11 sets the movement amount of the screen per basic operation based on the gesture to be larger than the movement amount of the screen per basic operation based on the touch. Here, the moving amount of the screen is, for example, a scrolling amount or the like, and means the magnitude of change in the display content of the screen. As described above, the amount of movement of the screen includes the amount of change in display content (for example, the number of skipped sheets) in an application or the like that displays a photograph. Further, the movement amount of the screen includes a change in coordinates when the map is scrolled, for example. Further, the moving amount of the screen includes, for example, an amount of change (for example, time for fast-forwarding) when fast-forwarding is instructed by an application for reproducing a moving image. The basic operation is a single operation. If it is a gesture, for example, it is an operation of moving one hand downward. Moreover, if it is a touch, it is one touch, for example.

  In the example of FIG. 21, the controller 11 sets the amount of screen movement S1 per basic operation based on touch, that is, the number of photos to be changed to one. Further, the controller 11 sets the screen movement amount S2 per basic operation based on a gesture (for example, an operation of moving the hand in the horizontal direction), that is, the number of photographs to be changed to ten. The controller 11 may store in the storage 16 the screen movement amount S1 per basic operation based on touch and the screen movement amount S2 per basic operation based on gestures.

  FIG. 22 is a diagram illustrating an example of an operation by touch in the seventh method. In the example of FIG. 22, when the first photograph is displayed on the display 14, the user performs, for example, one touch to move the touched finger leftward. When the touch is detected by the touch sensor 26, the controller 11 acquires the screen movement amount S <b> 1 per basic operation based on the touch from the storage 16. Then, the controller 11 changes the photograph for one sheet and causes the display 14 to display the second photograph. Here, the correspondence between the direction of the user's touch and the change in the photo number can be set arbitrarily. For example, when the user performs, for example, a touch that moves the touched finger downward, the photograph with the number incremented by one may be displayed on the display 14.

  FIG. 23 is a diagram illustrating an example of an operation by a gesture in the seventh method. In the example of FIG. 23, when the first photograph is displayed on the display 14, the user performs a gesture of movement that moves the hand, for example, rightward once. When the proximity sensor 18 detects a gesture, the controller 11 acquires a screen movement amount S2 per basic operation based on the gesture from the storage 16. Then, the controller 11 changes the 10 photos and displays the 11th photo on the display 14. Here, the correspondence between the direction of the user's gesture (for example, horizontal hand movement) and the change in the number of the photograph can be arbitrarily set. For example, when the user makes a leftward gesture once, a photograph with the number increased by 10 may be displayed on the display 14. Also, for example, when the user performs a leftward gesture once, a photograph with the number decreased by 10 may be displayed on the display 14.

  In the seventh method, as in the above example, an operation of moving the screen small by touching and an operation of moving the screen large by gesture are possible. Therefore, the user can move the screen larger than the touch by using the gesture (for example, by displaying a photograph with a distant number in the previous example), and for example, it is more efficient than when the touch is repeated many times. Operations can be performed. For example, in the example of FIG. 21, when displaying the twelfth photo, the user does not repeat twelve touches, but only performs one gesture and two touches. That is, the user can display desired data by moving the screen with fewer operations.

  Here, the controller 11 may perform the operation by the gesture only when a predetermined condition is satisfied. For example, in the above-described example of an application for displaying a photograph, when the photograph data is less than the movement amount (10 sheets) by one gesture, the controller 11 may be configured not to be able to perform the operation by the gesture. At this time, the controller 11 can turn off the proximity sensor 18 to suppress power consumption. In the example of the application for displaying the photo, the moving amount of the screen is in units of the number of photos, but may be set based on, for example, the time at the time of shooting. For example, the movement amount by the gesture can be set to one year. At this time, the user can display a photograph of one year ago or one year later by one gesture. At this time, if the difference in the photographing date and time of the photograph is less than the movement amount (one year) by one gesture, the controller 11 turns off the proximity sensor 18 so that the operation by the gesture cannot be performed. Good.

(Eighth method)
For example, when the electronic device 1 is operating in the car mode and the user wants to see a map around the current position of the vehicle, the user can move the screen by touching (eg, tapping) the touch panel display. . Here, the user may want to know information on an intersection (turning corner) away from the current position of the vehicle. However, particularly when the map is enlarged and displayed, the user may perform many touches until a desired intersection is displayed. For this reason, there has been a request to automatically display the next intersection. On the other hand, in such a navigation device, a function of small screen transition by a conventional touch is also necessary. The controller 11 of the electronic device 1 enables display of a screen with different movement amounts by associating gestures and touches with different information.

  FIG. 24 is a diagram illustrating an example of an operation by touch in the eighth technique. In the example of FIG. 24, a map is displayed on the display 14 (touch panel display). The figure P shows the current position of the vehicle on which the user rides. For example, when the user taps the touch panel display, the controller 11 acquires the coordinates of the tapped position from the touch panel display. Then, the controller 11 calculates the coordinates so that the tapped position becomes the center of the display 14, and moves the map based on the calculation result. As described above, in the eighth method, the controller 11 determines the amount of movement of the screen based on the coordinate information (an example of the second information) associated with the touch detected by the touch sensor 26. Here, as another example, the controller 11 may make the movement amount of the screen with one tap constant. For example, when the user taps a position located on the upper left side from the center of the touch panel display, the entire screen may be shifted in the opposite direction (downward right) by a certain amount of movement and displayed. Here, the fixed movement amount may be determined by a distance (for example, 500 meters or the like), for example.

  FIG. 25 is a diagram illustrating an example of an operation by a gesture in the eighth technique. The user performs a gesture of movement that moves the hand to the left, for example. When the proximity sensor 18 detects a gesture, the controller 11 acquires map information from the storage 16. The map information includes road and intersection information. Then, the controller 11 searches the next intersection I on the road that is on the left side of the current position of the vehicle and through which the vehicle is currently passing. Then, the controller 11 moves the map so that the position of the searched intersection I is displayed near the center of the display 14. Thus, in the eighth method, the controller 11 determines the amount of screen movement based on the intersection information (an example of the first information) associated with the gesture detected by the proximity sensor 18. Here, the controller 11 may display detailed information (for example, enlarged display) of the searched intersection I on another screen such as a pop-up. As another example, the controller 11 may search for the next intersection I in the current traveling direction of the vehicle regardless of the direction of the hand of the gesture.

  In the eighth method, as in the above example, the controller 11 associates the touch and the gesture with two different pieces of information (for example, coordinate information and intersection information). Then, the controller 11 sets different screen movement amounts based on two different pieces of information. The user moves the screen with a movement amount different from the touch by using the gesture (for example, by moving to the next intersection in the previous example), for example, an operation that is more efficient than when the touch is repeated many times. It can be performed.

(Ninth method)
For example, when the electronic device 1 is operating in a car mode and is involved in a traffic jam on a highway, the user may want to quickly know how far the traffic jam has continued. However, the conventional navigation device generally displays the names of subsequent ICs in order along with a background color indicating whether or not a traffic jam is occurring. Therefore, the user needs many touches in order to scroll and confirm an IC in which no traffic jam has occurred. The controller 11 of the electronic device 1 enables display of a screen with different movement amounts by associating gestures and touches with different information.

  FIG. 26 is a diagram illustrating an example of an operation by a gesture in the ninth method. The electronic device 1 displays information such as a map and an IC name displayed on the pop-up screen 144 on the display 14. For example, the user performs a movement gesture that moves the hand downward. When the proximity sensor 18 detects a gesture, the controller 11 acquires traffic information from the storage 16. Then, the controller 11 searches for the first IC (“GHI” in the example of FIG. 26) in which the traffic jam has been resolved among the ICs that the vehicle is scheduled to pass. Then, the controller 11 moves the display of the pop-up screen 144 so that the searched IC is displayed together with the previous two ICs (“ABC” and “DEF” in the example of FIG. 26). Here, “ABC” and “DEF” are displayed in a background color indicating that a traffic jam has occurred. Thus, in the ninth method, the controller 11 determines the amount of movement of the screen based on the traffic information (an example of the first information) associated with the gesture detected by the proximity sensor 18. Here, the user can also move the display of the pop-up screen by touching. At this time, information such as an IC name displayed on the pop-up screen 144 is shifted one by one.

  In the ninth method, similarly to the eighth method, the controller 11 associates the touch and the gesture with two different pieces of information (for example, IC information and traffic information). Then, the controller 11 sets different screen movement amounts based on two different pieces of information. By using the gesture, the user can move the screen with a movement amount different from that of the touch, and can perform an efficient operation compared to, for example, repeating the touch many times. Here, as another example, the controller 11 may use information on a service area or a parking area on a highway. For example, the controller 11 may immediately display a service area or a parking area that can be parked (not full) by the user's gesture based on the traffic information.

(flowchart)
FIG. 27 is a flowchart illustrating a process corresponding to the user operation of the electronic device 1 corresponding to the seventh technique.

  When the touch sensor 26 detects a touch (Yes in step S101), the controller 11 of the electronic device 1 moves the screen by the moving amount of the screen per basic operation based on the touch (step S102).

  When the touch sensor 26 does not detect a touch (No in step S101) or after the process in step S102, the controller 11 of the electronic device 1 proceeds to the process in step S103.

  When the proximity sensor 18 detects a gesture (Yes in step S103), the controller 11 of the electronic device 1 moves the screen by the movement amount of the screen per basic operation based on the gesture (step S104).

  The controller 11 of the electronic device 1 ends the series of processes when there is an end instruction for moving the screen after the process of step S104 (Yes in step S105).

  When the proximity sensor 18 does not detect a gesture (No in step S103) or when there is no end instruction for moving the screen (No in step S105), the controller 11 of the electronic device 1 returns to the process in step S101. .

  FIG. 28 is a flowchart illustrating processing corresponding to the user operation of the electronic apparatus 1 corresponding to the eighth method and the ninth method.

  When the touch sensor 26 detects a touch (Yes in step S201), the controller 11 of the electronic device 1 moves the screen by the screen movement amount based on the second information associated with the touch (step S202). ).

  When the touch sensor 26 does not detect a touch (No in step S201), or after the process in step S202, the controller 11 of the electronic device 1 proceeds to the process in step S203.

  When the proximity sensor 18 detects a gesture (Yes in step S203), the controller 11 of the electronic device 1 moves the screen by the amount of screen movement based on the first information associated with the gesture (step S204). ).

  The controller 11 of the electronic device 1 ends the series of processes when there is an end instruction for moving the screen after the process of step S204 (Yes in step S205).

  When the proximity sensor 18 does not detect a gesture (No in step S203) or when there is no end instruction for moving the screen (No in step S205), the controller 11 of the electronic device 1 returns to the process in step S201. .

  As described above, the controller 11 of the electronic device 1 of the present disclosure appropriately associates the gesture detected by the proximity sensor 18 and the touch detected by the touch sensor 26 with the operation. Therefore, the operability of the electronic device 1 of the present disclosure is improved with respect to the input operation.

(Other embodiments)
Although the present disclosure has been described based on the drawings and embodiments, it should be noted that various changes and modifications may be easily made by those skilled in the art based on the present disclosure. Accordingly, it should be noted that these variations and modifications are within the scope of the present disclosure. For example, functions included in each means or each step can be rearranged so as not to be logically contradictory, and a plurality of means or steps can be combined into one or divided. .

  For example, the above first to ninth methods can be combined. For example, each of the seventh to ninth techniques can be combined with the display form of the fifth technique having a plurality of screens.

  Moreover, the aspect in the case of displaying a plurality of screens is not limited to the case where the screen is divided by one display 14 as in the example of FIG. For example, when the electronic device 1 includes the display 14 on the front and back, the first screen 141 may be displayed on the display 14 on the back, and the second screen 142 may be displayed on the display 14 on the front.

  In the above embodiment, it has been described that the gesture is detected by the proximity sensor 18, but the gesture may not necessarily be detected by the proximity sensor 18. The gesture may be detected by any sensor capable of detecting the user's gesture without touching the device. An example of such a sensor includes a camera 13 and the like, for example.

  The sensor that can detect the user's gesture without touching the device itself may include, for example, a distance measuring sensor. For example, the electronic device 1 may include a distance measuring sensor instead of the proximity sensor 18 or together with the proximity sensor 18, and the gesture may be detected by the distance measuring sensor.

  The distance measuring sensor is a sensor that can measure the distance to the object. The distance measuring sensor may be constituted by, for example, a ToF (Time of Flight) sensor. A distance measuring sensor configured as a ToF sensor receives a light emitting unit that irradiates a target with sinusoidal modulated light (infrared laser light), and receives reflected light from the target of the irradiated infrared laser light. A light receiving unit. The light receiving unit includes, for example, an image sensor in which a plurality of light receiving elements are arranged. The ToF sensor measures time (flight time) from irradiation of infrared laser light to reception of reflected light by each light receiving element. The ToF sensor can measure the time of flight based on the phase difference between the irradiated infrared laser light and the received reflected light. The ToF sensor can measure the distance to the object reflecting the irradiated infrared laser beam based on the measured time of flight. The ToF sensor can detect the moving direction of the object based on the time difference in which the reflected light from the object enters each of the plurality of light receiving elements. Thereby, the ToF sensor can also detect a gesture performed by the user based on the same principle as that described for the proximity sensor 18. The distance measuring sensor may be disposed on the same surface as the surface on which the proximity sensor 18 is disposed, for example, in the electronic device 1.

  Here, a method in which the controller 10 detects the user's gesture based on the output of the distance measuring sensor will be described with reference to FIGS. 29 to 31. FIG. 29 is a diagram schematically showing the distance measuring sensor 126. FIG. 29 shows the distance measuring sensor 126 in a side view. The distance measuring sensor 126 includes a light emitting unit 126a and a light receiving unit 126b. It is assumed that the light emitting unit 126a and the light receiving unit 126b are arranged substantially parallel to the longitudinal direction of the electronic device 1. The light emitting unit 126a irradiates the target with infrared laser light. The light receiving unit 126b receives reflected light from the object of the irradiated infrared light.

  The light receiving unit 126b may include a plurality of light receiving elements. For example, the light receiving unit 126b may include nine light receiving elements arranged in 3 rows × 3 columns as shown in FIG. Each of the nine light receiving elements receives reflected light from the object. In the upper part of the light receiving unit 126b, three light receiving elements Ch11, Ch12, and Ch13 are arranged in order from the left, approximately parallel to the short direction of the electronic device 1. In the light receiving unit 126b, three light receiving elements Ch21, Ch22, and Ch23 are arranged in the middle from the left in substantially parallel to the short direction of the electronic device 1. In the lower part of the light receiving unit 126b, three light receiving elements Ch31, Ch32, and Ch33 are arranged in order from the left, substantially parallel to the short direction of the electronic device 1.

  The distance measuring sensor 126 is a distance from each of the nine light receiving elements to the object based on the phase difference between the infrared laser light emitted by the light emitting unit 126a and the reflected light received by each of the nine light receiving elements of the light receiving unit 126b. Can be measured. The distance measuring sensor 126 can detect a gesture based on the distance from each of the nine light receiving elements to the object and the change in the distance over time.

  For example, assume that the user performs a gesture of moving his hand from left to right. At this time, for example, the distances to the object detected by the light receiving elements Ch21, Ch22, and Ch23 in the middle stage are D21, D22, and D23, respectively. FIG. 31 is a diagram schematically showing the transition of the distance to the object detected by each light receiving element. For example, as schematically shown in FIG. 31, first, the hand that is the object approaches the light receiving element Ch21 that is first arranged on the left side, and therefore the distance D21 of the object detected by the light receiving element Ch21 is reduced. Thereafter, when the hand, which is the object, approaches the light receiving element Ch22 disposed in the center, the distance D22 of the object detected by the light receiving element Ch22 is reduced. Finally, when the hand that is the object moves to the right side, the distance D23 of the object that is detected by the light receiving element Ch23 arranged on the right side becomes shorter. The order in which the hands approaching the light receiving elements Ch21, Ch22, and Ch23 move away is also the order of Ch21, Ch22, and Ch23. Therefore, the distances D21, D22, and D23 increase in this order (return to the initial value). The vertical gesture can also be detected by using the light receiving elements Ch12, Ch22, and Ch32 arranged in the longitudinal direction based on the same principle. As described above, the distance measuring sensor 126 can detect a gesture based on the distance from each of the nine light receiving elements to the object and the change in the distance with time.

  Here, the light receiving unit 126b is described as including nine light receiving elements, but the number of light receiving elements included in the light receiving unit 126b is not limited thereto. The arrangement of the plurality of light receiving elements is not limited to the arrangement shown in FIG. The number and arrangement of the light receiving elements included in the light receiving unit 126b may be appropriately determined according to the type of gesture to be detected.

  The light emitting unit 126a of the distance measuring sensor 126 may include a plurality of light emitting elements. In this case, the distance from each of the nine light emitting elements to the object can be measured based on the phase difference between the infrared laser light emitted from each light emitting element and the reflected light received by the light receiving unit 126b. Even in this case, the distance measuring sensor 126 can detect a gesture by applying the above-described principle based on the distance from each of the nine light emitting elements to the object and the change in the distance over time. it can.

  Many aspects of the present disclosure are presented as a series of operations performed by a computer system or other hardware capable of executing program instructions. Examples of the computer system and other hardware include a general-purpose computer, a PC (personal computer), a dedicated computer, and a workstation. Computer systems and other hardware include PCS (Personal Communications System), mobile (cellular) telephones, and mobile telephones with data processing functions. Computer systems and other hardware also include RFID receivers, game consoles, electronic notepads and laptop computers. The computer system and other hardware also include a GPS (Global Positioning System) receiver and other programmable data processing devices. In each embodiment, the various operations or control methods are performed by dedicated circuitry (eg, individual logic gates interconnected to perform a specific function), eg, implemented with program instructions (software). Please keep in mind. It should be noted that in each embodiment, various operations or control methods are executed by, for example, logical blocks and / or program modules executed by one or more processors. The one or more processors that execute logic blocks and / or program modules include, for example, one or more microprocessors and a CPU (Central Processing Unit). Such a processor includes, for example, an application specific integrated circuit (ASIC) and a digital signal processor (DSP). Such a processor includes, for example, a PLD (Programmable Logic Device) and an FPGA (Field Programmable Gate Array). Such processors also include controllers, microcontrollers, microprocessors, electronics, and other devices designed to perform the functions described herein. Such processors also include combinations of the above specific examples. The embodiments shown here are implemented by, for example, hardware, software, firmware, middleware, microcode, or any combination thereof. The instructions may be program code or code segments for performing necessary tasks. The instructions can then be stored on a machine-readable non-transitory storage medium or other medium. A code segment may represent any combination of procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes or instructions, data structures or program statements. A code segment transmits and / or receives information, data arguments, variables or stored contents with other code segments or hardware circuits, thereby connecting the code segments with other code segments or hardware circuits .

  The storage 16 used here can be further configured as a computer-readable tangible carrier (medium) composed of solid state memory, magnetic disk and optical disk. Such a medium stores an appropriate set of computer instructions or a data structure such as a program module for causing a processor to execute the technology disclosed herein. Computer readable media include electrical connections with one or more wires, magnetic disk storage media, magnetic cassettes, magnetic tape, and other magnetic storage devices. The computer-readable medium includes an optical storage device (for example, a CD (Compact Disk) and a laser disk (registered trademark). The computer-readable medium includes a DVD (registered trademark) (Digital Versatile). Disc), floppy (registered trademark) disc, and Blu-ray disc (registered trademark)). The computer-readable medium includes a portable computer disk, a random access memory (RAM), and a read-only memory (ROM). The computer readable medium includes EPROM (Erasable Programmable Read-Only Memory). The computer readable medium includes EEPROM (Electrically Erasable Programmable Read-Only Memory). The computer readable medium also includes a rewritable and programmable ROM such as a flash memory or other tangible storage medium capable of storing information, or a combination of any of the above specific examples. The memory can be provided inside and / or outside the processor or processing unit. As used herein, the term “memory” means any kind of long-term storage, short-term storage, volatile, non-volatile or other memory. That is, the “memory” is not limited to a specific type and / or number. Further, the type of medium in which the storage is stored is not limited.

DESCRIPTION OF SYMBOLS 1 Electronic device 11 Controller 12 Timer 13 Camera 14 Display 15 Microphone 16 Storage 17 Communication unit 18 Proximity sensor 19 UV sensor 20 Illuminance sensor 21 Acceleration sensor 22 Geomagnetic sensor 23 Air pressure sensor 24 Gyro sensor 25 Speaker 26 Touch sensor 30 Icon 31 Thumbnail 126 Measurement Distance sensor 126a Light emitting portion 126b Light receiving portion 141 First screen 142 Second screen 143 Third screen 180 Infrared LED for light source
181 Lens SU, SR, SD, SL Photodiode

Claims (7)

A sensor that detects a gesture without touching its own device,
A touch sensor;
A display for displaying the first screen and the second screen in a multi-window ;
Depending on the position where the sensor is located, Ru and a controller for associating each touch detected by the gesture and the touch sensor to the operation and the operation of the second screen of the first screen is detected by the sensor Electronics. The controller is
The electronic device according to claim 1, wherein the gesture is associated with an operation on the first screen closer to the second screen than a position where the sensor is disposed, and the touch is associated with an operation on the second screen. The controller is
The electronic device according to claim 1, wherein the touch is associated with the operation on the second screen by associating the gesture with the operation on the first screen farther than the second screen from the position where the sensor is arranged. The controller is
The electronic device according to claim 1, wherein the touch is further associated with at least one of an operation between screens and a screen erasing operation. The controller is
A specific touch that is one of the touches that can be executed on the second screen is associated with a specific operation on the second screen,
5. The electronic device according to claim 1, wherein one of the gestures is associated with the specific operation on the first screen. 6. A control method for an electronic device comprising: a sensor that detects a gesture without touching the device itself; a touch sensor; and a display that displays the first screen and the second screen in a multi-window ,
Depending on the position where the sensor is located, the control method comprising the steps of associating each touch detected by the gesture and the touch sensor is detected in the operation of the operation and the second screen of the first screen by the sensor . An electronic device comprising a sensor that detects a gesture without touching the device itself, a touch sensor, and a display that displays the first screen and the second screen in multiple windows .
Depending on the position where the sensor is located, the program for executing the steps of associating each touch detected by the gesture and the touch sensor is detected in the operation of the operation and the second screen of the first screen by the sensor .

Images