JP6259225B2 - Electronic device, gesture recognition operation method for mobile terminal connected to the same, and in-vehicle system - Google Patents

Description

  The present invention relates to an electronic device, a gesture recognition operation method for a mobile terminal connected to the electronic device, and an in-vehicle system, and in particular, displays an image displayed on the screen of the mobile terminal on the electronic device and operates the mobile terminal from the electronic device. Therefore, the present invention is suitable for application to a system that can be used.

  In recent years, a technique called a mirror link (registered trademark: the same applies below) that links a mobile terminal such as a smartphone and an in-vehicle device has been provided. Mirror link is a mechanism that displays the image displayed on the mobile device as it is on the display of the in-vehicle device like a mirror, so that the mobile terminal can be operated through the image displayed on the in-vehicle device display. Yes. As a result, an application corresponding to a mirror link installed in the mobile terminal (hereinafter referred to as a mobile application) can be operated on the display of the in-vehicle device.

  For example, as shown in FIG. 4, a map image being displayed on the mobile terminal 101 is transmitted from the mobile terminal 101 installed with the map application to the in-vehicle device 102, and the same map as the mobile terminal 101 is displayed on the display of the in-vehicle device 102. It is possible to display an image. In addition, position information (coordinate information) representing a touch position on the touch panel of the in-vehicle device 102 is transmitted from the in-vehicle device 102 to the mobile terminal 101, and the mobile terminal 101 is mounted on the touch panel of the mobile terminal 101 based on this position information. It is also possible to perform the same processing (such as map scrolling, enlargement / reduction, rotation, etc.) as if the same position as the device 102 was touch-operated.

  As described above, the mirror link is configured to transmit the touch position on the touch panel of the in-vehicle device as coordinate information from the in-vehicle device to the mobile terminal so that the mobile application can be operated on the display of the in-vehicle device. Therefore, information for operating the mobile application based on a command obtained by recognizing a gesture in space performed by the user on the in-vehicle device side cannot be transmitted to the mobile terminal. Therefore, in order to operate the mobile app using gesture recognition in the in-vehicle device, it is necessary to make the mobile app itself compatible with gesture recognition and use another communication control that is not a mirror link.

  In addition, when operating a mobile phone with a touch panel that supports multi-touch with a touch panel in-vehicle device that supports only single touch, the type of multi-touch gesture (for example, tap operation using two fingers or 2 A technology that sends an input operation signal of the selected gesture operation to a mobile phone by performing a pseudo operation with a single touch after selecting a pinch operation that expands or contracts the distance between the fingers. Has been proposed (see, for example, Patent Document 1).

JP 2012-8968 A

  However, with the technique described in Patent Document 1, it is only possible to generate an input operation signal for a multi-touch gesture by a pseudo operation of a single touch gesture. For this reason, there is a problem in that it is impossible to recognize a gesture performed in a space away from the touch panel and operate the mobile terminal for an image transmitted and displayed from the mobile terminal to the in-vehicle device.

  The present invention has been made to solve such a problem, and allows an electronic device to display an image displayed on a screen of a mobile terminal so that the mobile device can be operated from the electronic device. Another object of the present invention is to enable an operation on a touch panel of a portable terminal based on recognition of a gesture in space in an electronic device.

In order to solve the above-described problem, in the present invention, an electronic device configured to receive and display image data generated by a mobile terminal with a touch panel and to operate the mobile terminal, A conversion table storage unit that stores conversion table information for converting a gesture input item to be recognized for a gesture performed in space by a user into a coordinate point or a coordinate string representing a gesture movement of the electronic device-side touch panel; When the user gesture recognized by the gesture recognition unit corresponds to the gesture input item, the conversion table information is used to generate a coordinate point or a coordinate sequence that represents the movement of the gesture on the electronic device side touch panel from the gesture input item. I am trying to send it.

  According to the present invention configured as described above, when a user gesture recognized by an electronic device corresponds to a gesture input item, the gesture input item is converted into a predetermined coordinate point or coordinate sequence and transmitted to the mobile terminal. Therefore, the mobile terminal can perform the same processing as when the position corresponding to the coordinate point or the coordinate string is operated on the touch panel of the mobile terminal based on the coordinate point or the coordinate string. Thereby, operation on the touch panel of a portable terminal can be performed based on gesture recognition with an electronic device.

It is a block diagram which shows the function structural example of the vehicle equipment which concerns on one Embodiment of the electronic device of this invention. It is a figure which shows an example of the conversion table information stored in the conversion table memory | storage part by this embodiment. It is a flowchart which shows the operation example of the vehicle equipment by this embodiment. It is a figure for demonstrating operation | movement of the vehicle equipment cooperated with the mirror link, and the portable terminal.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a functional configuration example of an in-vehicle device according to an embodiment of an electronic device of the present invention. The in-vehicle device 10 of the present embodiment cooperates with the mobile terminal 20 with a touch panel by a mirror link. Then, the image data generated by the mobile terminal 20 is received and displayed, and an operation (map scroll, enlargement / reduction, rotation) of an application (a map application in this embodiment) installed in the mobile terminal 20 is performed. ) Can be performed from the vehicle-mounted device 10.

  As shown in FIG. 1, the in-vehicle device 10 of the present embodiment includes an image data receiving unit 11, an image display control unit 12, a conversion table storage unit 13, a captured image input unit 14, a gesture recognition unit 15, as its functional configuration. An input item determination unit 16 and a control unit 17 are provided. Here, the in-vehicle device 10 is connected to a display 30 with a touch panel (hereinafter also simply referred to as a display 30) and a camera 40, or is configured integrally.

  Each of the functional blocks 11 to 12 and 14 to 17 can be configured by any of hardware, DSP (Digital Signal Processor), and software. For example, when configured by software, each of the functional blocks 11 to 12 and 14 to 17 is actually configured by including a CPU, a RAM, a ROM, and the like of a computer included in the in-vehicle device 10, and the RAM, the ROM, the hard disk, or the semiconductor memory. This is realized by operating a program stored in a recording medium such as the above.

  The image data receiving unit 11 receives image data relating to a map image being displayed on the display 21 with a touch panel of the mobile terminal 20 (hereinafter also simply referred to as the display 21) from the mobile terminal 20. The image display control unit 12 displays a map image on the display 30 based on the image data received by the image data receiving unit 11.

  The display 21 of the mobile terminal 20 is a vertically long type, and the display 30 connected to the in-vehicle device 10 is a horizontally long type. When the mobile terminal 20 is mirror-linked to the in-vehicle device 10, the mobile terminal 20 converts the vertically long map image being displayed on the display 21 into a horizontally long map image to be displayed on the display 30 of the in-vehicle device 10, and After changing the display on the display 21 to a horizontally long map image, the converted image data is transmitted to the in-vehicle device 10. The image data receiving unit 11 of the in-vehicle device 10 receives this image data.

  When image data is received from the portable terminal 20 by the image data receiving unit 11 and a map image is displayed on the display 30 based on the image data by the image display control unit 12, the position where the map image is displayed on the display 30 (For example, coordinate information of the four corners of the display 30) is returned from the in-vehicle device 10 to the portable terminal 20. The portable terminal 20 stores the coordinate information of the four corners of the display 30 returned from the in-vehicle device 10 and the coordinate information of the four corners of the display 21 in association with each other (hereinafter, the stored information is referred to as “coordinate association information”). ).

  Thereby, coordinate information representing the touch position of the in-vehicle device 10 with respect to the touch panel is transmitted from the in-vehicle device 10 to the mobile terminal 20, and the mobile terminal 20 uses the coordinate information on the touch panel of the mobile terminal 20 based on the coordinate association information. By converting to, it becomes possible to perform the same processing as when the same position as the vehicle-mounted device 10 is touch-operated on the touch panel of the mobile terminal 20.

  The conversion table storage unit 13 converts a plurality of gesture input items desired to be gesture recognition into coordinate points or coordinate strings (hereinafter collectively referred to as coordinate information) on the touch panel of the display 30 with a touch panel. Table information is stored in advance. In the present embodiment, the gesture input items include those related to at least one of scrolling, enlarging / reducing, and rotating the map image.

  For example, the conversion table storage unit 13 converts the gesture input items related to “right scroll”, “left scroll”, “upward scroll”, and “downward scroll” into predetermined coordinate sequences for scrolling the map image. Information is stored. Here, the gesture input item related to “right scroll” indicates an operation of moving the hand from right to left in the space, and a command “movement of moving the hand from right to left” indicating the operation. Information is stored. Corresponding to this gesture input item, information on a coordinate string related to “right scroll” is stored.

  Similarly, the gesture input items related to “left scroll”, “up scroll”, and “down scroll” are each an action of moving the hand from left to right in space, and moving the hand from top to bottom in space. It shows an action, an action of moving a hand from below to above in space, and command information indicating each action is stored. Corresponding to these gesture input items, information on coordinate strings related to “left scroll”, “upward scroll”, and “downward scroll” is stored.

  In addition, the conversion table storage unit 13 stores information for converting the gesture input items relating to “enlargement” and “reduction” into predetermined coordinate strings for the enlargement / reduction of the map image. Here, the gesture input item relating to “enlargement” indicates an operation of expanding the interval between two fingers in the space, and command information indicating the operation is stored. Similarly, the gesture input item relating to “reduction” indicates an operation of reducing the interval between two fingers in the space, and command information indicating the operation is stored. Corresponding to these gesture input items, information on coordinate strings relating to “enlargement” and “reduction” is stored.

  Furthermore, the conversion table storage unit 13 stores information for converting the gesture input items related to “right rotation” and “left rotation” into predetermined coordinate strings with respect to the rotation of the map image. Here, the gesture input item relating to “right rotation” indicates an operation of rotating the hand to the right in the space, and command information indicating the operation is stored. Similarly, the gesture input item regarding “left rotation” indicates an operation of rotating the hand to the left in the space, and command information indicating the operation is stored. Corresponding to these gesture input items, information of coordinate strings related to “right rotation” and “left rotation” is stored.

  FIG. 2 is a diagram showing conversion table information for converting “right scroll” and “enlarged” gesture input items into predetermined coordinate strings, as an example. As shown in FIG. 2, the conversion table storage unit 13 stores a predetermined length from the right to the left from the predetermined start position to the end position on the touch panel corresponding to the “right scroll” gesture input item. A coordinate sequence obtained when a flick operation is performed with only one finger is stored. Also, in response to the “enlarge” gesture input item, when a pinch operation is performed using two fingers to widen the interval around a predetermined start position on the touch panel (for example, the center point of the touch panel) The obtained coordinate sequence is stored.

  The captured image input unit 14 inputs a captured image of the user's finger from the camera 40. For example, the camera 40 is set in the vehicle-mounted device 10 or in the vicinity thereof, and images a predetermined range in front of the vehicle-mounted device 10. The gesture recognition unit 15 recognizes a finger gesture (hereinafter referred to as a user gesture) performed in space by the user from the captured image input by the captured image input unit 14. Then, command information representing the recognized gesture is output.

  The input item determination unit 16 refers to the conversion table information stored in the conversion table storage unit 13 and determines whether or not the user gesture recognized by the gesture recognition unit 15 corresponds to a gesture input item. That is, the input item determination unit 16 determines whether or not the command information related to the user gesture output from the gesture recognition unit 15 is stored in the conversion table storage unit 13 as command information of the gesture input item.

  When the input item determination unit 16 determines that the user gesture recognized by the gesture recognition unit 15 corresponds to a gesture input item, the control unit 17 uses the conversion table information stored in the conversion table storage unit 13. Predetermined coordinate information is generated from the gesture input item and transmitted to the mobile terminal 20.

  In the portable terminal 20, based on the coordinate information sent from the in-vehicle device 10 by the control unit 17, the same processing as when the position corresponding to the coordinate information is operated on the touch panel of the portable terminal 20 is performed. That is, in the mobile terminal 20, the coordinate information (pseudo operation coordinate information on the touch panel on the touch panel-equipped display 30) sent from the in-vehicle device 10 is used as the touch panel of the mobile terminal 20 using the coordinate association information described above. The pseudo operation coordinate information is converted to the above, and the same processing as that in which the position corresponding to the coordinate information after the conversion is operated is performed.

  FIG. 3 is a flowchart showing an operation example of the vehicle-mounted device 10 configured as described above according to the present embodiment. Note that the processing of the flowchart shown in FIG. 3 starts when the in-vehicle device 10 and the mobile terminal 20 are connected and linked by a mirror link. First, the image data receiving unit 11 receives image data related to a map image being displayed on the display 21 of the mobile terminal 20 from the mobile terminal 20 (step S1). And the image display control part 12 displays a map image on the display 30 based on the image data received by the image data receiving part 11 (step S2). Thereafter, the in-vehicle device 10 returns the coordinate information of the four corners of the display 30 on which the map image is displayed to the mobile terminal 20 (step S3).

  Next, the captured image input unit 14 determines whether or not a captured image of the user is input from the camera 40 (step S4). When the captured image is not input, the in-vehicle device 10 determines whether or not the cooperation with the mobile terminal 20 by the mirror link is finished (step S5), and when the cooperation is finished, the process of the flowchart shown in FIG. Exit. If the cooperation has not ended, the process returns to step S1. Thereby, as the map image is updated in the mobile terminal 20, the map image is also updated and displayed on the in-vehicle device 10 as needed.

  On the other hand, when a captured image is input by the captured image input unit 14, the gesture recognition unit 15 recognizes a user gesture from the captured image input by the captured image input unit 14 (step S6). Next, the input item determination unit 16 determines whether or not the user gesture recognized by the gesture recognition unit 15 corresponds to a predetermined gesture input item (step S7). Here, when it is determined that the user gesture recognized by the gesture recognition unit 15 does not correspond to the predetermined gesture input item, the input item determination unit 16 notifies the user that the user gesture does not correspond to the gesture input item. A message is displayed on the display 30 (step S8). Thereafter, the process returns to step S1.

  On the other hand, when the input item determination unit 16 determines that the user gesture recognized by the gesture recognition unit 15 corresponds to the gesture input item, the control unit 17 uses the conversion table information stored in the conversion table storage unit 13. Then, predetermined coordinate information is generated from the gesture input item and transmitted to the portable terminal 20 (step S9). Thereafter, the process returns to step S1. Thereby, a predetermined process is executed in the mobile terminal 20 based on the coordinate information sent from the in-vehicle device 10, and the same map image as the updated map image is displayed on the display 30.

  As described above in detail, the vehicle-mounted device 10 of the present embodiment includes the conversion table storage unit 13 that stores conversion table information for converting gesture input items for gesture recognition into predetermined coordinate information, and includes a gesture recognition unit. When the user gesture recognized by 15 corresponds to a gesture input item, predetermined coordinate information is generated from the gesture input item using the conversion table information and transmitted to the mobile terminal 20.

  According to the present embodiment configured as described above, when a user gesture recognized by the in-vehicle device 10 corresponds to a gesture input item, the gesture input item is converted into predetermined coordinate information and transmitted to the mobile terminal 20. Therefore, the mobile terminal 20 can perform the same processing as when the position corresponding to the coordinate information is operated on the touch panel of the mobile terminal 20 based on the coordinate information. Thereby, operation on the touch panel of the portable terminal 20 can be performed based on the gesture recognition in the vehicle-mounted device 10.

  In the above embodiment, the time for transmitting the coordinate information from the in-vehicle device 10 to the portable terminal 20 may be controlled according to the speed of the gesture performed by the user (time when the gesture is performed). For example, in addition to the content of the user gesture, the gesture recognition unit 15 also recognizes the time when the user gesture is performed (the time from the start to the end of the user gesture).

  When the input item determination unit 16 determines that the user gesture recognized by the gesture recognition unit 15 corresponds to a gesture input item, the control unit 17 coordinates at a speed corresponding to the time recognized by the gesture recognition unit 15. Information is transmitted to the mobile terminal 20. For example, the coordinate information is transmitted to the portable terminal 20 over the same time (or time proportional to) the time when the user gesture is performed. As a result, the mobile terminal 20 executes processing such as scrolling of the map image at a speed matching the time when the coordinate information is sent, and the map image displayed on the in-vehicle device 10 is also scrolled at the same speed. Done.

  Moreover, in the said embodiment, you may enable it to designate the starting position of the coordinate information produced | generated by the control part 17. FIG. For example, in addition to the content of the user gesture, the gesture recognition unit 15 also recognizes the position on the display 30 indicated by the user's finger by the user gesture. And the control part 17 produces | generates the coordinate information according to the content of the gesture input item from the position recognized by the gesture recognition part 15, and transmits to the portable terminal 20. FIG.

  Moreover, although the map application was mentioned and demonstrated in the said embodiment as an example of the portable application installed in the portable terminal 20, the portable application which can apply this invention is not limited to a map application. That is, any mobile application having a function of executing a predetermined process in response to a touch operation or a tracing operation on the touch panel can be applied.

  Further, in the above embodiment, a predetermined coordinate sequence is generated according to a user gesture corresponding to a tracing operation such as flick, pinch, and rotation, and is transmitted from the in-vehicle device 10 to the mobile terminal 20, thereby scrolling and enlarging the image. Although an example of executing processing such as reduction and rotation has been described, the present invention is not limited to this. For example, even when a predetermined coordinate point is generated according to a user gesture corresponding to a tap operation with one or two fingers and transmitted from the in-vehicle device 10 to the mobile terminal 20, thereby executing a predetermined process. The present invention can be applied.

  In the above embodiment, the in-vehicle device 10 is described as an example of the electronic device. However, the electronic device to which the present invention is applicable is not limited to the in-vehicle device 10. That is, any electronic device compatible with a mirror link or a communication control technique similar to this can be applied.

  In addition, each of the above-described embodiments is merely an example of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the gist or the main features thereof.

10 Onboard equipment (electronic equipment)
DESCRIPTION OF SYMBOLS 11 Image data receiving part 12 Image display control part 13 Conversion table memory | storage part 14 Captured image input part 15 Gesture recognition part 16 Input item determination part 17 Control part 20 Portable terminal

Claims (6)

The electronic device is configured to receive and display image data generated by a mobile terminal with a touch panel, and to operate the mobile terminal,
An electronic device-side touch panel that accepts user touch operations;
A conversion table storage unit storing conversion table information for converting a gesture input item to be recognized for a gesture performed in space by the user into a coordinate point or a coordinate string representing the movement of the gesture on the electronic device-side touch panel When,
A captured image input unit for inputting the user's captured image;
A gesture recognition unit for recognizing a user gesture that is a gesture performed on the space by the user from a captured image input by the captured image input unit;
An input item determination unit that determines whether or not a user gesture recognized by the gesture recognition unit corresponds to the gesture input item;
When it is determined by the input item determination unit that the user gesture recognized by the gesture recognition unit corresponds to the gesture input item, the gesture of the gesture on the electronic device side touch panel is determined from the gesture input item using the conversion table information . An electronic apparatus comprising: a control unit that generates a coordinate point or a coordinate sequence representing movement and transmits the generated coordinate point or coordinate sequence to the portable terminal. The gesture recognition unit recognizes the time when the user gesture is performed in addition to the content of the user gesture,
When the input item determination unit determines that the user gesture recognized by the gesture recognition unit corresponds to the gesture input item, the control unit moves the gesture from the gesture input item using the conversion table information. 2. The electronic apparatus according to claim 1, wherein a coordinate point or a coordinate sequence representing the position is generated, and the coordinate point or the coordinate sequence is transmitted to the portable terminal at a speed according to a time recognized by the gesture recognition unit. machine. The gesture recognition unit recognizes the position on the display of the electronic device indicated by the user's finger by the user gesture in addition to the content of the user gesture,
When the input item determination unit determines that the user gesture recognized by the gesture recognition unit corresponds to the gesture input item, the control unit starts from the position recognized by the gesture recognition unit as the starting point. The electronic device according to claim 1, further comprising: a control unit that generates a coordinate point or a coordinate sequence representing the movement of the gesture from the gesture input item using information and transmits the generated coordinate point or coordinate sequence to the mobile terminal.   The electronic device according to claim 1, wherein the gesture input item includes a gesture input item related to at least one of scrolling, enlargement / reduction, and rotation of an image. In a system configured to display an image displayed on a mobile terminal with a touch panel on an electronic device having an electronic device-side touch panel that accepts a user's touch operation, the mobile device uses gesture recognition from the electronic device. A gesture recognition operation method for operating a terminal,
A first step in which a captured image input unit of the electronic device inputs a captured image of a user;
A second step in which a gesture recognition unit of the electronic device recognizes a user gesture which is a gesture performed in space by the user from the captured image input by the captured image input unit;
A third step in which the input item determination unit of the electronic device determines whether the user gesture recognized by the gesture recognition unit corresponds to a predetermined gesture input item;
Control unit of the electronic device, the case where the user gesture recognized by the gesture recognition unit is determined by the corresponding to the gesture input item the input item determining unit, the gesture input items of the electronic apparatus side panel above gesture A fourth step of generating a coordinate point or a coordinate string representing the movement of the gesture from the gesture input item using the conversion table information for converting into a coordinate point or a coordinate string representing the movement of the character and transmitting it to the portable terminal A gesture recognition operation method for a portable terminal connected to an electronic device. An in-vehicle system configured to display an image displayed on a mobile terminal with a touch panel on an in-vehicle device and to operate the mobile terminal from the in-vehicle device,
The in-vehicle device is
An electronic device-side touch panel that accepts user touch operations;
A conversion table storage unit storing conversion table information for converting a gesture input item to be recognized for a gesture performed in space by the user into a coordinate point or a coordinate string representing the movement of the gesture on the electronic device-side touch panel When,
A captured image input unit for inputting the user's captured image;
A gesture recognition unit for recognizing a user gesture that is a gesture performed on the space by the user from a captured image input by the captured image input unit;
An input item determination unit that determines whether or not a user gesture recognized by the gesture recognition unit corresponds to the gesture input item;
When it is determined by the input item determination unit that the user gesture recognized by the gesture recognition unit corresponds to the gesture input item, the gesture of the gesture on the electronic device side touch panel is determined from the gesture input item using the conversion table information . A control unit that generates a coordinate point or a coordinate sequence representing movement and transmits the generated coordinate point or coordinate sequence to the mobile terminal,
The portable terminal performs the same processing as when the position corresponding to the coordinate sequence is traced on the touch panel based on the coordinate sequence sent from the in-vehicle device by the control unit. In-vehicle system.

Images