WO2014098164A1

WO2014098164A1 - Image display device, mobile terminal device, information processing system, control method for image display device and program

Info

Publication number: WO2014098164A1
Application number: PCT/JP2013/084005
Authority: WO
Inventors: 英徳塚原
Original assignee: 日本電気株式会社
Priority date: 2012-12-21
Filing date: 2013-12-19
Publication date: 2014-06-26
Also published as: JP5871080B2; JPWO2014098164A1

Abstract

　Provided is an image display device which, without requiring a specific application program for remote operation, allows for remote operation of a mobile terminal and improves workability. The image display device is provided with: a touch operation detecting unit for detecting a touch operation on a display screen of the display device; a cursor operation information generating unit for generating, on the basis of the touch operation and conforming to a predetermined communication system, a cursor movement instruction signal and a click instruction signal; a cursor operation information transmitting unit for transmitting the cursor movement instruction signal and the click instruction signal in accordance with the predetermined communication system; and a calibrating unit for acquiring calibration information so that a position specified by the touch operation on the display screen of the display device and a position of the cursor moved on the basis of the cursor movement instruction signal on a display screen of a mobile terminal device are relatively identical positions, using an image displayed on the display device and on the mobile terminal device as reference.

Description

Image display device, portable terminal device, information processing system, image display device control method, and program

The present invention relates to an image display device that controls a mobile terminal device such as a smartphone, a mobile terminal device that performs a predetermined calibration process with the image display device, an information processing system, an image display device control method, and a program.

In recent years, information terminal devices have become multifunctional with the progress of communication technology, and portable terminal devices such as smartphones and tablet terminals have become widespread. The mobile terminal device has not only a function for sending and receiving e-mail and the like, but also a browser function for acquiring a document file described in a predetermined description language from a web server connected to the Internet and browsing a web page as standard. . In such portable terminal devices, in addition to the push-type button input, those having touch panel means that can be input by touching the screen with a finger or the like are mainly used.

In-vehicle information terminal devices such as car navigation systems (hereinafter referred to as “car navigation”) installed in automobiles, etc., send and receive e-mail using Internet communication in addition to the existing route search function and TV viewing function. Those that enable collection of various information such as browsing homepages and electronic commerce have been released.

By the way, many users who own automobiles have portable terminal devices having a function of transmitting and receiving data using wireless communication together with in-vehicle information terminal devices mounted on automobiles. In order to use the Internet communication in the in-vehicle information terminal device, a communication function is required, but it is difficult for the user to pay a separate communication cost for the in-vehicle information terminal device such as a car navigation system because it is expensive. Therefore, a mobile terminal device owned by a user and an in-vehicle information terminal device are connected via an interface such as a wireless LAN (Local Area Network), Bluetooth (registered trademark), or a wired cable, and the communication function of the mobile terminal device is used. Thus, it is common to provide a communication function for the in-vehicle information terminal device.

Here, Patent Literature 1 discloses an in-vehicle information terminal device that communicates with a mobile terminal device.
There is also known a technique that enables a mobile terminal device having a navigation function to be operated by an in-vehicle information terminal device. In this technology, a mobile terminal device is caused to execute an application function for remote operation for enabling remote operation from the in-vehicle information terminal device.

Japanese Laid-Open Patent Publication No. 2012-141001

In the technology that enables operation of the mobile terminal device with the in-vehicle information terminal device, a dedicated application program for remote operation from the in-vehicle information terminal device is required in the mobile terminal device, and the dedicated application program is installed in the mobile terminal device The work of doing was essential. In addition, the mobile terminal device needs to operate while sequentially analyzing information from the in-vehicle information terminal device by operating a dedicated application program for remote operation from the in-vehicle information terminal device on the own mobile terminal device. The followability of the in-vehicle information terminal device with respect to the operation information is poor and the operability is lowered.

Therefore, an object of the present invention is to provide an image display device, a portable terminal device, an information processing system, an image display device control method, and a program that can solve the above-described problems.

The present invention has been made to solve the above-described problem, and includes an image display unit that displays an image displayed on a display screen of a mobile terminal device, and a touch operation that detects a touch operation on the display screen of the self-image display device. A detection unit; a cursor operation information generation unit that generates a cursor movement instruction signal and a click instruction signal in accordance with a predetermined communication method based on the touch operation; and the cursor movement instruction signal and the click instruction signal. Based on the cursor movement instruction signal on the display screen of the portable terminal device, the position designated by the touch operation on the display screen of the self-image display device, The position of the moved cursor and the image displayed by the self-portrait display device and the mobile terminal device, respectively. A calibration unit for obtaining calibration information for the same relative position Te is an image display device comprising a.

The present invention also relates to an image display device that displays an image displayed on the display screen of its own mobile terminal device and a mobile terminal device that performs a predetermined calibration process, wherein a cursor movement instruction signal that conforms to a predetermined communication method is received. A control unit that receives and moves the cursor on the display screen of the portable terminal device based on the cursor movement instruction signal, and the control unit further includes a part of the display screen of the portable terminal device. When a predetermined click instruction signal is received from the image display device when a predetermined response area is set in the response area and the cursor is in the response area, the portable terminal device transmits a predetermined response signal. is there.

Further, the present invention is an information processing system including an image display device that displays an image displayed on a display screen of a mobile terminal device, the image display device, and a mobile terminal device that performs a predetermined calibration process, The image display device generates a touch operation detection unit that detects a touch operation on the display screen of the self-image display device, and a cursor movement instruction signal and a click instruction signal that comply with a predetermined communication method based on the touch operation. A cursor operation information generation unit, a cursor operation information transmission unit that transmits the cursor movement instruction signal and the click instruction signal according to the predetermined communication method, and the touch operation specified on the display screen of the self-image display device A position and a cursor moved on the display screen of the mobile terminal device based on the cursor movement instruction signal A calibration unit for acquiring calibration information for making the positions relatively the same with reference to images displayed by the image display device and the mobile terminal device, respectively, the calibration unit further comprising: The transmission of the cursor movement instruction signal and the transmission of the click instruction signal following the transmission of the cursor movement instruction signal are repeated a plurality of times via the cursor operation information transmission unit, and the mobile terminal device for the click instruction signal is transmitted. The calibration information is acquired based on the presence or absence of a predetermined response signal, and the portable terminal device receives the cursor movement instruction signal, and based on the cursor movement instruction signal, the portable terminal device A control unit that moves the cursor on the display screen; and the control unit is further provided on a part of the display screen of the mobile terminal device. Set the response region of the constant, in the case where the cursor was in the response area, upon receiving the click instruction signal is an information processing system for transmitting the response signal.

Further, the present invention displays an image displayed on the display screen of the mobile terminal device on another display screen, a position touched on the other display screen displaying the image, and the display of the mobile terminal device The position of the cursor moved based on a predetermined cursor movement instruction signal on the screen, and the image displayed on the other display screen and the image displayed on the display screen of the portable terminal device as a reference The calibration information for obtaining the relatively same position is acquired, the touch operation on the other display screen is detected, and the cursor movement instruction signal and the click instruction signal are generated based on the touch operation. In the image display device control method, the cursor movement instruction signal and the click instruction signal are transmitted according to a predetermined communication method.

The present invention also provides a computer of an image display device that displays an image displayed on a display screen of a mobile terminal device, a touch operation detection unit that detects a touch operation on the display screen of the self-image display device, and the touch operation based on the touch operation. A cursor operation information generation unit that generates a cursor movement instruction signal and a click instruction signal in accordance with a predetermined communication method; a cursor operation information transmission unit that transmits the cursor movement instruction signal and the click instruction signal according to the predetermined communication method The position designated by the touch operation on the display screen of the self-portrait display device and the position of the cursor moved based on the cursor movement instruction signal on the display screen of the mobile terminal device, respectively And the same position relative to the image displayed by the mobile terminal device Calibration unit for obtaining calibration information because a program to function as a.

In the present invention, it becomes possible to remotely operate the portable terminal device without requiring a dedicated application program for remote operation, and an effect of improving operability can be obtained.

It is a figure which shows the minimum structure of the image display apparatus in one Embodiment of this invention. It is a figure which shows concretely the function structure of the image display apparatus and portable terminal device in one Embodiment of this invention. It is a 1st figure explaining operation | movement of the image display apparatus and portable terminal device in one Embodiment of this invention. It is a 2nd figure explaining operation | movement of the image display apparatus and portable terminal device in one Embodiment of this invention. It is a 3rd figure explaining operation | movement of the image display apparatus and portable terminal device in one Embodiment of this invention. It is a figure explaining the content of the cursor movement instruction | indication signal in one Embodiment of this invention concretely. It is a figure explaining the content of the cursor movement instruction | indication signal in one Embodiment of this invention concretely. It is a figure which shows the movement amount multiplication rate information which the portable terminal device in one Embodiment of this invention has. It is a figure explaining the initial operation | movement of the calibration part in one Embodiment of this invention. It is a 1st figure explaining the 1st specific process of the calibration part in one Embodiment of this invention. It is a 1st figure which shows progress of the process of the calibration part in one Embodiment of this invention. It is a figure explaining the 2nd specific process of the calibration part in one Embodiment of this invention. It is a 2nd figure which shows progress of the process of the calibration part in one Embodiment of this invention. It is a 2nd figure explaining the 1st specific process of the calibration part in one Embodiment of this invention. It is a 3rd figure which shows progress of the process of the calibration part in one Embodiment of this invention. It is a 1st figure which shows the processing flow of the calibration part and portable terminal device in one Embodiment of this invention. It is a 2nd figure which shows the processing flow of the calibration part and portable terminal device in one Embodiment of this invention. It is a figure which demonstrates concretely operation | movement of the cursor operation information generation part in one Embodiment of this invention.

Hereinafter, an image display apparatus according to an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing a minimum configuration of the image display apparatus according to the embodiment. In this figure, reference numeral 1 denotes an image display device.

As shown in FIG. 1, the image display device 1 includes at least a touch operation detection unit 100, a cursor operation information generation unit, a calibration unit 121, and a cursor operation information transmission unit 130.
The touch operation detection unit 100 is a functional unit that detects a touch operation on the display screen of the image display device 1.
The cursor operation information generation unit 120 is a functional unit that generates a cursor movement instruction signal and a click instruction signal conforming to a predetermined communication method based on the touch operation (the cursor movement instruction signal and the click instruction signal will be described later). ).
The cursor operation information transmission unit 130 is a functional unit that transmits the cursor movement instruction signal and the click instruction signal according to a predetermined communication method.
The calibration unit 121 includes a position designated by a touch operation on the display screen of the self-portrait display device, and a position of the mouse cursor moved based on the cursor movement instruction signal on the display screen of the mobile terminal device 2 (FIG. 2). Is a functional unit that obtains calibration information for making the positions relatively the same with reference to images displayed by the respective devices.

FIG. 2 is a diagram specifically illustrating a functional configuration of the image display device and the mobile terminal device according to the embodiment of the present invention.
As illustrated in FIG. 2, the image display device 1 includes a touch panel 10, a display unit 11, a control unit 12, a wireless communication unit 13, and a wired communication unit 14.

The touch panel 10 is a panel having a predetermined touch sensor, and is a functional unit that forms a touch sensor display together with the display unit 11. On the other hand, the display unit 11 is a flat display configured by a liquid crystal display, an organic EL (Electro Luminescence) display, and the like, and is a functional unit that transmits various types of information through a user's vision.

The touch panel 10 according to the present embodiment is realized by using a capacitance method, a resistance method, or the like, as in a commonly used touch sensor display, and the position touched on the display screen of the display unit 11 is represented by coordinate information ( X, Y). The touch panel 10 is an aspect of the touch operation detection unit 100 in FIG.

Further, the display unit 11 in the present embodiment displays an image that is based on the image information received by the wired communication unit 14 from the mobile terminal device 2 and is equivalent to the image displayed on the display screen of the mobile terminal device 2. . For example, when an application function for car navigation (hereinafter, “car navigation function”) is executed in the mobile terminal device 2 and an image for the car navigation function is displayed on the display screen (display unit 21), The display unit 11 of the image display device 1 also displays the image for the car navigation function.

The control unit 12 is a functional unit that executes various processes in accordance with information input from various functional units in addition to the touch panel 10 and outputs the process or result of the processing to a predetermined functional unit. The control unit 12 is generally configured by causing a CPU (Central Processing Unit, central processing unit) to execute a predetermined program. In the present embodiment, it is assumed that the cursor operation information generation unit 120 and the calibration unit 121 illustrated in FIG. Specific functions of the cursor operation information generation unit 120 and the calibration unit 121 will be described later.

The wireless communication unit 13 is a functional unit that performs wireless communication with the mobile terminal device 2 in accordance with a predetermined communication method. Specifically, the wireless communication unit 13 according to the present embodiment implements wireless communication using a Bluetooth (registered trademark) HID (Human Interface Device) profile (hereinafter simply referred to as “HID profile”). Here, Bluetooth (registered trademark) is one of short-range wireless communication standards for electronic devices. The HID profile is a wireless communication standard for an input device such as a mouse or a keyboard among Bluetooth (registered trademark) profiles.
The wireless communication unit 13 is an aspect of the cursor operation information transmission unit 130 illustrated in FIG. The “HID profile” is an aspect of the “predetermined communication method” in the present embodiment. That is, the wireless communication unit 13 in the present embodiment transmits a cursor movement instruction signal, a press instruction signal, or a release instruction signal that is mouse operation information based on the HID profile (the contents of each instruction signal will be described later). The wireless communication unit 13 may transmit and receive a keyboard operation signal based on the HID profile in addition to the cursor movement instruction signal, the press instruction signal, or the release instruction signal.
In this embodiment, the exchange of mouse operation information between the image display device 1 and the portable terminal device 2 is realized using wireless communication (Bluetooth (registered trademark) HID profile) as described above. However, other embodiments are not limited to this aspect. That is, when transmitting / receiving mouse operation information, the mouse operation information may be transmitted / received using predetermined wired communication assuming a USB (Universal Serial Bus) mouse.

The wireless communication unit 13 performs wireless communication using a Bluetooth (registered trademark) SPP (Serial Port Profile) profile (hereinafter simply referred to as “SPP profile”). The SPP profile is a profile for converting an electronic device that communicates with Bluetooth (registered trademark) into a virtual serial port. The wireless communication unit 13 receives, for example, the image resolution on the display unit 21 of the mobile terminal device 2 and the orientation (vertical, horizontal) of the mobile terminal device 2 by wireless communication compliant with the SPP profile.

The wired communication unit 14 is a functional unit that performs wired communication with the mobile terminal device 2. The wired communication unit 14 is a wired connection unit having, for example, an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal. The wired communication unit 14 is connected to the wired communication unit 24 of the mobile terminal device 2 via the HDMI cable 3. The wired communication unit 14 in the present embodiment receives image information of an image being displayed on the display unit 21 of the mobile terminal device 2, for example.
In the present embodiment, the exchange of image information between the image display device 1 and the mobile terminal device 2 is realized using wired communication (HDMI), but is limited to this aspect in other embodiments. It is not a thing. That is, when a sufficient communication speed can be realized for transmitting and receiving image information, the image information may be transmitted and received using wireless communication.

On the other hand, the mobile terminal device 2 includes a touch panel 20, a display unit 21, a control unit 22, a wireless communication unit 23, and a wired communication unit 24.

The basic functions of the touch panel 20 and the display unit 21 are the same as the functions of the touch panel 10 and the display unit 11 included in the image display device 1. As described above, for example, when the car navigation application function is executed in the mobile terminal device 2, the display unit 21 displays an image for the car navigation application function.

The control unit 22 included in the mobile terminal device 2 is a functional unit that performs various processes in the mobile terminal device 2, similarly to the control unit 12 in the image display device 1. The control unit 22 is configured by causing the CPU to execute a predetermined program in the same manner as the control unit 12. The control unit 22 is a functional unit that reads out an application program describing a car navigation function, for example, and realizes the car navigation function.

Further, the control unit 22 receives a cursor movement instruction signal conforming to the HID profile, and moves the mouse cursor on the screen of the display unit 21 based on the cursor movement instruction signal.
For example, when the control unit 22 receives a “cursor movement instruction signal” as mouse operation information from an external device, the control unit 22 displays the mouse cursor displayed on the display unit 21 of the mobile terminal device from the current position. The process of moving in the distance and direction based on the is executed. The cursor movement instruction signal is a signal based on the HID profile that is output by the mouse itself in response to an operation of moving the mouse by the user.
In addition, when receiving a “press instruction signal” as mouse operation information from the external device, the control unit 22 executes processing based on a mouse button press operation (an operation of pressing a predetermined button provided on the mouse). Similarly, when receiving a “release instruction signal” as mouse operation information, the control unit 22 executes processing based on a mouse button release operation (an operation to release the pressed button).
As a generally known mouse operation, there is a “click operation” in which a release operation is performed after a press operation. Further, a “drag operation” is performed by moving the mouse cursor while maintaining the press operation. In the present embodiment, a series of signal sets of release instruction signals following the press instruction signal is defined as a “click instruction signal”.

The wireless communication unit 23 establishes wireless communication based on the HID profile and the SPP profile with respect to the wireless communication unit 13 of the image display device 1. For example, the wireless communication unit 23 receives mouse operation information (cursor movement instruction signal, press instruction signal, release instruction signal) based on the HID profile from the wireless communication unit 13 of the image display apparatus 1.
Similar to the wired communication unit 14 of the image display device 1, the wired communication unit 24 is a wired connection unit having an HDMI terminal, and performs wired communication with the wired communication unit 14 via the HDMI cable 3. For example, the wired communication unit 24 transmits image information of an image being displayed on the display unit 21 to the wired communication unit 14 of the image display device 1.

In summary, the image display device 1 according to the present embodiment establishes wired communication and wireless communication with the mobile terminal device 2 in which the car navigation function is executed, and functions as a car navigation system mounted on an automobile or the like. Device. The image display device 1 according to the present embodiment is an image display device that displays an image displayed on the display screen of the mobile terminal device 2, and the operation of a user performing a car navigation function realized in the mobile terminal device 2. It can be said that it is an auxiliary device used for the purpose of improving convenience.
However, in this case, the mobile terminal device 2 is a device that functions as the basis of the car navigation system, while the image display device 1 uses the image displayed on the display screen of the mobile terminal device 2 as the display screen of the self-image display device. And a user interface device that reflects an operation based on the user's intention on the display image as an operation on the mobile terminal device 2. That is, the image display device 1 can function as an aspect other than the car navigation system depending on the application function executed in the mobile terminal device 2. For example, when the mobile terminal device 2 is executing a video playback function, the image display device 1 functions as a video playback system by establishing wired communication and wireless communication with the mobile terminal device 2 that has executed the video playback function. .

FIG. 3 is a first diagram illustrating operations of the image display device and the mobile terminal device according to the embodiment of the present invention.
Next, specific operations of the image display device 1 and the mobile terminal device 2 that function as a car navigation system will be described. First, the portable terminal device 2 that has executed the application program for car navigation is connected to the image display device 1 by the HDMI cable 3. Then, the control unit 22 (not shown in FIG. 3) of the portable terminal device 2 transmits image information of the image (main image A) being displayed on the display unit 21 of the own portable terminal device via the wired communication unit 24. To do. The main image A is a display image for realizing the car navigation function in the mobile terminal device 2.

Then, the control unit 12 (not shown in FIG. 3) of the image display device 1 displays the main image A ′ based on the image information of the main image A received via the wired communication unit 14 as a display unit of the self-image display device. 11 is output. Note that the control unit 12 simultaneously receives the resolution information for the main image A. The resolution information is information indicating the aspect ratio and resolution of an image based on the number of pixels W2 in the horizontal direction and the number of pixels H2 in the vertical direction of the display unit 21 of the mobile terminal device 2.

Here, as shown in FIG. 3, the control unit 12 enlarges and displays the main image A to the limit that the entire main image A ′ based on the received main image A can be drawn on the display unit 11. Since the display (display unit 21) used for the ordinary portable terminal device 2 is small, it is difficult for the user to visually recognize the information as it is, but the image display device 1 displays the main image A ′ as the self-image display device. This problem can be solved by displaying the entire display unit 11.

However, at this time, depending on the model of the mobile terminal device 2, the aspect ratio of the display unit 21 of the mobile terminal device 2 and the display unit 11 of the image display device 1 may not match. Therefore, the control unit 12 displays the main image A ′ on the display unit 11 of the self-image display device while maintaining the aspect ratio of the main image A, and the area of the display screen of the display unit 11 where the main image A ′ is not displayed. A blank image B (for example, a black image) is displayed (FIG. 3). For example, the display unit 21 of the mobile terminal device 2 has a resolution of “W2 = 600, H2 = 400” (both are the number of pixels), while the display unit 11 of the image display device 1 has “W1 = 900, H1 = 700. ”. In this case, the control unit 12 displays the image obtained by enlarging the main image A by 1.5 times (= 900/600) as the main image A ′ on the display unit 11 with a resolution of “W1 = 900, H1 ′ = 600”. Further, the margin image B is displayed in the upper and lower margin portions (the region corresponding to the width obtained by subtracting H1 ′ = 600 from H1 = 700).

FIG. 4 is a second diagram illustrating operations of the image display device and the mobile terminal device according to the embodiment of the present invention.
Hereinafter, the operation of the cursor operation information generation unit 120 when a touch operation (operation in which a finger is brought into contact with the touch panel 10 and a finger is released) is detected on the touch panel 10 of the image display device 1 will be described in detail. The cursor operation information generation unit 120 generates mouse operation information (cursor movement instruction signal, press instruction signal, release instruction signal) based on the HID profile based on the user's touch operation on the touch panel 10 of the image display device 1. And the said mouse operation information is produced | generated so that the touch operation to the image display apparatus 1 based on a user's intention may be reflected as a mouse operation in the portable terminal device 2. FIG. Here, the cursor operation information generation unit 120 assumes that the mouse cursor in the mobile terminal device 2 is in advance at the coordinates (0, 0) that is the reference position (for example, the lower right corner on the display unit 21). In particular, a cursor movement instruction signal) is generated.

Here, it is assumed that the touch panel 10 first detects a touch operation at coordinates (X1, Y1) (right side in FIG. 4). Then, the cursor operation information generation unit 120 immediately generates a cursor movement instruction signal (X2, Y2) corresponding to the touch position (X1, Y1). Here, the cursor operation information generation unit 120 generates a cursor movement instruction signal based on the scale of the display unit 21 of the mobile terminal device 2 with respect to the display unit 11 of the self-portrait display device acquired in advance and predetermined “calibration information”. (“Calibration information” will be described later). Then, the cursor operation information generation unit 120 outputs the cursor movement instruction signal (X2, Y2) and then the click instruction signal (press instruction signal and subsequent release instruction signal) to the wireless communication unit 13.

Here, the “touch time” of the touch operation performed by the user, that is, the time from when the finger touches the touch panel 10 until it is released is from when the press instruction signal is transmitted in the click instruction signal until the release instruction signal is transmitted. May correspond to the time of That is, the cursor operation information generation unit 120 outputs a press instruction signal by detecting that the finger touches the touch panel 10, and then outputs a release instruction signal by detecting that the finger has left the touch panel 10. May be. In this way, a so-called “long press” operation performed by the user (an operation that keeps touching the touch panel 10 for a certain period of time) can be reflected as a mouse operation in the mobile terminal device 2.

The control unit 22 (not shown in FIG. 4) of the mobile terminal device 2 receives the cursor movement instruction signal (X2, Y2) and the click instruction signal via the wireless communication unit 23. Here, the control unit 22 receives the various received signals as mouse operation information based on the HID profile. Then, the control unit 22 moves the cursor in accordance with the cursor movement instruction signal (X2, Y2) from the current position (reference position (0, 0)) to the X axis direction (left direction on the paper) by the X2 width and in the Y axis direction (paper surface). Move upward Y) by Y2 width (left side in FIG. 4). Then, based on the click instruction signal received subsequently, the control unit 22 executes a process when a mouse click operation is received.

The cursor operation information generation unit 120 generates and outputs a series of mouse operation information as described above, so that the touch operation on the image display device 1 based on the user's intention is performed in the portable terminal device 2. It is processed as a mouse operation reflecting the user's intention (moving the mouse cursor and clicking). The cursor operation information generation unit 120 outputs a cursor movement instruction signal for returning the mouse cursor to the reference position (0, 0) after outputting a cursor movement instruction signal and a click instruction signal based on the touch operation. In this way, the cursor operation information generation unit 120 moves the cursor on the assumption that the cursor position of the mobile terminal device 2 is at the reference position (0, 0) every time a touch operation is detected in the self-image display device. An instruction signal can be generated.

FIG. 5 is a third diagram for explaining the operation of the image display device and the mobile terminal device according to the embodiment of the present invention.
Next, the operation of the cursor operation information generation unit 120 when a drag operation (operation in which a finger touches the touch panel 10 and slides to another position as it is) is detected on the touch panel 10 of the image display device 1 will be described in detail. To do. The cursor operation information generation unit 120 generates mouse operation information based on the HID profile based on the user's drag operation, as in the case of the touch operation.

First, when the cursor operation information generation unit 120 detects the start of the drag operation (the finger touching the touch panel 10), the cursor movement instruction signal (X2, Y2) corresponding to the initial position (X1, Y1) of the drag operation. Is generated. This process is equivalent to the cursor movement process for the touch operation process described above. That is, the cursor operation information generation unit 120 generates the cursor movement instruction signal (X2, Y2) on the assumption that the mouse cursor in the mobile terminal device 2 is in the reference position (0, 0) in advance. Further, the cursor operation information generation unit 120 generates a press instruction signal following the cursor movement instruction signal and transmits the press instruction signal via the wireless communication unit 13.

Next, it is assumed that the touch panel 10 detects a drag operation from the coordinates (X1, Y1) to the coordinates (X1 ′, Y1 ′) (right side in FIG. 5). Then, the cursor operation information generation unit 120 calculates the difference (dX1, dY1) between the start point position (X1, Y1) and the end point position (X1 ′, Y1 ′) of the drag operation, and according to the difference coordinates (dX1, dY1). The cursor movement instruction signal (dX2, dY2) is generated. Also in this case, the cursor operation information generation unit 120 generates a cursor movement instruction signal based on the scale of the display unit 21 with respect to the display unit 11 and predetermined calibration information (described later).

The control unit 22 (not shown in FIG. 5) of the mobile terminal device 2 receives the cursor movement instruction signal (dX2, dY2) via the wireless communication unit 23. Here, the control unit 22 receives the received signal as mouse operation information based on the HID profile. Then, in accordance with the cursor movement instruction signal (dX2, dY2), the control unit 22 moves the mouse cursor from the current position (X2, Y2) to the X-axis direction (leftward on the paper) by the dX2 width and in the Y-axis direction (downward on the paper) ) By dY2 width (left side in FIG. 5).

Also, the cursor operation information generation unit 120 does not output the release instruction signal until it detects the end of the drag operation (the finger has been released from the touch panel 10) after outputting the press instruction signal at the start of the drag operation. The mouse button is kept pressed. By doing in this way, the cursor operation information generation part 120 can make the drag operation using the finger in the image display device 1 correspond to the drag operation using the mouse in the mobile terminal device 2. When the end of the drag operation is detected, the cursor operation information generation unit 120 outputs a cursor movement instruction signal for returning the mouse cursor to the reference position (0, 0) following the release instruction signal. By doing in this way, the cursor operation information generation part 120 can return the cursor position of the portable terminal device 2 to the reference position (0, 0) even after the end of the drag operation.

As described above, the image display device 1 in the present embodiment reproduces the user's “touch operation” and “drag operation” on the image display device 1 as mouse operations on the mobile terminal device 2. When such an image display device 1 is used, the user does not need to directly operate a small display (display unit 21) of the mobile terminal device 2, and a relatively large size display (display unit 11) of the image display device 1 is used. ) To operate the mobile terminal device 2 as intended.

In the present embodiment, as described above, the mobile terminal device 2 is remotely operated using an HID profile that can be normally used in the case of a general mobile terminal device 2. Therefore, the user does not need to install an application program for cooperative operation for causing the image display device 1 and the mobile terminal device 2 to function as a car navigation system. Further, if the HID profile is used, the mobile terminal device 2 receives a cursor movement instruction signal, a press instruction signal, and a release instruction signal as signals using a normal wireless mouse. The processing is also executed promptly as an operation using a normal wireless mouse.

In the image display device according to the present embodiment described above, it is possible to operate the mobile terminal device 2 from the image display device 1 without requiring an application program for remote operation from the image display device 1. The effect which improves property is acquired.

By the way, as described above, in order for the touch operation and the drag operation in the image display device 1 to be reproduced as the mouse operation in the mobile terminal device 2 as intended by the user, at the touch position or the drag operation in the display unit 11. The designated position and the position where the mouse cursor is moved on the display unit 21 based on the cursor movement instruction signal must always be relatively the same with respect to the image displayed on each display unit. .

However, in reality, the aspect ratio and resolution of the display unit 21 vary depending on the mobile terminal device 2 used by the user. Further, even if the image display apparatus 1 transmits a cursor movement instruction signal having the same content, depending on the specification (or user setting) of the mobile terminal apparatus 2 that has received the signal, the movement in which the mouse cursor actually moves on the display unit 21 The amount will vary. Therefore, the image display device 1 does not prepare a predetermined piece of calibration information, but every time the mobile terminal device 2 is connected to the image display device 1, the image display device 1 is used for calibration of the mobile terminal device 2. Need to get information.

Therefore, prior to the processing of the cursor operation information generation unit 120 described above, the calibration unit 121 and the position specified by the touch operation on the display screen of the display unit 11 and the display screen of the display unit 21 of the mobile terminal device 2 are displayed. The calibration information necessary to make the position of the mouse cursor moved based on the cursor movement instruction signal always the same position on the basis of the image displayed on each display unit is acquired.

6A and 6B are diagrams for specifically explaining the contents of the cursor movement instruction signal according to the embodiment of the present invention.
First, before describing the specific content of the calibration process in the calibration unit 121, the content of the cursor movement instruction signal will be described in detail with reference to FIGS. 6A and 6B. 6A and 6B show cursor movement instruction signals having different contents.

Here, the cursor movement instruction signal based on the HID profile is composed of a combination of two pieces of information, “movement amount L” and “movement speed V”. The movement amount L is a reference amount for moving the mouse cursor described above. In the case of a general wireless mouse, the value of the movement amount L transmitted from the wireless mouse corresponds to the distance that the user actually moved the wireless mouse. That is, when the user greatly moves the wireless mouse, the movement amount L received by the receiving side (mobile terminal device 2) increases in accordance with the movement distance. However, on the receiving side, the received movement amount L is not applied as it is as the movement amount of the mouse cursor on the display unit 21. This point will be described later.

When the receiving side receives “Lx = 100” for the movement amount Lx in the X-axis direction, the receiving side performs a process of moving the mouse cursor from the current position in accordance with “Lx = 100” in the X-axis direction. The same applies to the Y-axis direction, and the movement amount L is represented by a combination of a movement amount Lx in the X-axis direction and a movement amount Ly in the Y-axis direction.

Further, the transmission side of a cursor movement instruction signal such as a general wireless mouse divides the movement amount L into a predetermined divided movement amount Lp, and transmits this divided into a plurality of times. Then, the receiving side receives the divided movement amount Lp per transmission as “movement speed V” which is another information. For example, the cursor movement instruction signals shown in FIGS. 6A and 6B are recognized as the movement amount “L = 30” for the receiving side. However, the cursor movement instruction signal shown in FIG. 6A sets the movement amount L to “30” by transmitting the divided movement amount Lp per transmission as “15” in two steps. On the other hand, in the cursor movement instruction signal shown in FIG. 6B, the divided movement amount Lp per transmission is set to “7.5”, and the movement amount L is set to “30” by dividing the transmission into four times.

The receiving side receives the movement amount per unit time (for example, 1 second) based on the divided movement amount Lp as the movement speed V. For example, in the case of FIG. 6A, the receiving side receives “90”, which is the movement amount per unit time based on the divided movement amount “Lp = 15” per transmission, as the movement speed V. Similarly, in the case of FIG. 6B, “45”, which is the movement amount per unit time based on the divided movement amount “Lp = 7.5” per transmission, is received as the movement speed V.
Similarly to the movement amount L, the movement speed V is represented by a combination of a movement speed Vx in the X-axis direction and a movement speed Vy in the Y-axis direction.

FIG. 7 is a diagram illustrating movement amount multiplying rate information included in the mobile terminal device according to the embodiment of the present invention.
The control unit 22 of the mobile terminal device 2 determines the actual movement amount of the mouse cursor on the display unit 21 based on the movement amount L and the movement speed V defined as described above. Here, the control unit 22 specifies a “multiplication rate” by which the movement amount received by referring to “movement amount multiplication rate information” set in advance in the mobile terminal device is multiplied. That is, the control unit 22 executes the movement process using the value obtained by multiplying the received movement amount by the “multiplication rate” as the actual movement amount of the mouse cursor.
The range of moving speed V is defined in the left column of the moving amount multiplying rate information table shown in FIG. 7, and the multiplying rate corresponding to each range of moving speed V is defined in the right column. That is, the multiplication rate to be applied is specified based on the moving speed V to be received.

For example, it is assumed that the mobile terminal device 2 receives a movement amount “L = 100” and a movement speed “V = 200” as a cursor movement instruction signal from the HID profile transmission side. Then, the control unit 22 specifies the multiplication rate “A = 3” corresponding to the movement speed “V = 200” with reference to the movement amount multiplication rate information table (FIG. 7). Then, the received movement amount “L = 100” is multiplied by “A = 3” to calculate the movement amount “LA = 300” after the multiplication rate is applied. The control unit 22 performs a process of moving the mouse cursor by the width of the movement amount “LA = 300” after application of the multiplication rate on the display unit 21 (hereinafter, LA is referred to as “effective movement amount”).

In the example shown in FIG. 6A (movement amount “L = 30”, movement speed “V = 90”), the effective movement amount LA that is the actual movement amount of the mouse cursor on the display unit 21 is “39” ( In the example shown in FIG. 6B (movement amount “L = 30”, movement speed “V = 45”), the effective movement amount LA is “30” (= 30 × 1) (see the table in FIG. 7). Thus, even if the movement amount L is the same, the mobile terminal device 2 performs a process of increasing the effective movement amount LA as the movement speed V increases.

The process in which the control unit 22 of the mobile terminal device 2 dynamically changes the effective movement amount LA according to the movement speed V is a known technique used to make normal mouse operations performed by the user comfortable. is there. That is, when the user quickly moves the mouse, the mobile terminal device 2 automatically increases the amount of movement of the mouse cursor on the assumption that the user intends to move the cursor greatly. On the other hand, when the user moves the mouse slowly, it is assumed that the user intends to fine-tune the cursor, and the amount of movement of the mouse cursor is automatically reduced.
Note that the movement amount adjustment function according to the movement speed as described above is used not only for the mobile terminal device 2 but also for all electronic devices that accept mouse operations.

However, the information on how much the moving amount increases according to how much moving speed, that is, the moving amount multiplying rate information table shown in FIG. 7 is unique information determined in the mobile terminal device 2, and It can also be customized according to user settings.
Therefore, in order for the image display device 1 to realize the above-described cooperation processing with the mobile terminal device 2 (FIGS. 4 and 5), the image display device 1 has the movement amount multiplying rate information inherently possessed by the mobile terminal device 2. Need to get. The cursor operation information generation unit 120 associates the touch coordinates (X1, Y1) in the touch operation with the movement amounts (X2, Y2) of the mouse cursor in the mobile terminal device 2 while referring to the acquired movement amount multiplication rate information. I do.

Therefore, the calibration unit 121 of the image display device 1 according to the present embodiment performs a predetermined calibration process, and is a unique information determined in the mobile terminal device 2 and is applied to the movement amount L applied according to the movement speed V. The moving amount multiplying rate information defining the multiplying rate is specified based on the processing procedure shown below. In the present embodiment, the movement amount multiplying rate information is an aspect of “calibration information”.
On the other hand, the control unit 22 of the mobile terminal device 2 reads a predetermined calibration application program in order to realize the calibration process of the calibration unit 121. Then, the control unit 22 that has read the calibration application program sets a predetermined response area 211 in a part of the screen of the display unit 21, and when the mouse cursor is in the response area 211, a click instruction signal Is received, a process of transmitting a predetermined response signal is executed.
Hereinafter, specific means for acquiring the movement amount multiplication rate information in the calibration unit 121 will be described.

FIG. 8 is a diagram illustrating the initial operation of the calibration unit according to the embodiment of the present invention.
Here, it is assumed that the number of pixels of the display unit 21 of the mobile terminal device 2 in this embodiment is 580 in the X-axis direction (horizontal direction on the paper surface) and 350 in the Y-axis direction (vertical direction on the paper surface).
First, the calibration unit 121 transmits an activation instruction signal for the calibration application function via the wireless communication unit 13 prior to the start of the calibration process. On the other hand, when the control unit 22 (not shown in FIG. 8) of the mobile terminal device 2 receives the activation instruction signal via the wireless communication unit 23, the control unit 22 reads a calibration application program stored in advance in the mobile terminal device 2. To execute a predetermined calibration application function. Hereinafter, a procedure in which the calibration unit 121 acquires the movement amount multiplication rate information in the X-axis direction using the calibration application function will be described.

FIG. 8 shows a state where a display image based on the calibration application function is displayed on the display unit 21. The display image based on the calibration application function includes a non-response area 210 and a response area 211 as shown in FIG. The response area 211 is set to an area of one pixel width (at the left end of the display unit 21 in FIG. 8) farthest from the reference position (0, 0) of the mouse cursor. The non-response area 210 is set as all areas other than the response area 211.

When the mouse cursor is located in the response area 211, the control unit 22 receives a click instruction signal from the outside, and transmits a predetermined response signal via the wireless communication unit 23 based on the calibration application function. On the other hand, if the control unit 22 receives a click instruction signal from the outside when the mouse cursor is positioned in the non-response area 210, the control unit 22 does not transmit the predetermined response signal.

The calibration unit 121 repeats the transmission of the cursor movement instruction signal and the subsequent transmission of the click instruction signal via the wireless communication unit 13 a plurality of times, and receives a predetermined response signal from the mobile terminal device 2 in response to the click instruction signal. A calibration process is executed based on the presence or absence.

FIG. 9 is a first diagram illustrating a first specifying process of the calibration unit according to the embodiment of the present invention.
The calibration unit 121 first performs a “first identification process” for identifying the multiplication rate A (the right column in FIG. 7) corresponding to the moving speed Vx.
The calibration unit 121 first transmits a predetermined cursor movement instruction signal to move the mouse cursor to the reference position (0, 0). Here, the calibration unit 121 can move the mouse cursor of the mobile terminal device 2 to the reference position (0, 0) even when the image display device 1 does not grasp the movement amount multiplication rate information. is there. For example, if the calibration unit 121 transmits a sufficiently large amount of movement with respect to the resolution of the display unit 21 of the mobile terminal device 2, the mouse cursor naturally fits at the end of the display unit 21 (that is, the reference position).

Next, the calibration unit 121 repeats transmission of the cursor movement instruction signal and the click instruction signal a plurality of times while increasing the movement amount Lx by a predetermined amount while the movement speed Vx is fixed to a predetermined maximum movement speed. Here, the maximum movement speed is a movement speed corresponding to the maximum value of the divided movement amount Lp that can be sent in one transmission in the HID profile. In the following description, this maximum moving speed is set to 300 as an example.

First, in FIG. 9A, the calibration unit 121 transmits a cursor movement instruction signal with a movement amount “Lx = 100” and a movement speed “Vx = 300” (maximum movement speed). The mobile terminal device 2 receives the cursor movement instruction signal and moves the mouse cursor based on the received contents (movement amount “Lx = 100”, movement speed “Vx = 300”). At this point, since the multiplication rate A in the mobile terminal device 2 is unknown, the image display device 1 cannot grasp how much the position of the mouse cursor moved in the mobile terminal device 2 has actually moved from the reference position. .

The calibration unit 121 then transmits a click instruction signal. When the mobile terminal device 2 receives the click instruction signal, the mobile terminal device 2 executes a process when a click operation using a mouse is received. Here, since the mouse cursor in the mobile terminal device 2 belongs to the non-response area 210 as shown in FIG. 9A, a predetermined response signal is not transmitted in response to the click instruction signal. The image display device 1 detects that the response signal is not received for a predetermined period, and grasps that the mouse cursor has not reached the response area 211 (that is, the end of the display unit 21).

Next, the calibration unit 121 transmits a cursor movement instruction signal for once returning the position of the mouse cursor to the reference position. After the mouse cursor is returned to the reference position, the calibration unit 121 increases the movement amount Lx from the previous time while keeping the movement speed Vx fixed at “300”, and transmits the cursor movement instruction signal again as “Lx = 101”. To do. The mobile terminal device 2 receives the cursor movement instruction signal and moves the mouse cursor. When the calibration unit 121 detects that there is no response signal for a predetermined period with respect to the click instruction signal transmitted subsequently, the calibration unit 121 repeats the same processing while further increasing the movement amount Lx.

FIG. 9B shows a state after the calibration unit 121 repeats the process of transmitting the cursor movement instruction signal while increasing the movement amount Lx a plurality of times. In FIG. 9B, the calibration unit 121 transmits a cursor movement instruction signal with the movement amount “Lx = 290” and the movement speed “Vx = 300”. The mobile terminal device 2 receives the cursor movement instruction signal and moves the mouse cursor based on the received contents (movement amount “Lx = 290”, movement speed “Vx = 300”).

The calibration unit 121 then transmits a click instruction signal. When the mobile terminal device 2 receives the click instruction signal, the mobile terminal device 2 executes a process when a click operation using a mouse is received. Here, since the mouse cursor in the mobile terminal device 2 has reached the response area 211 as shown in FIG. 9B, the mobile terminal device 2 transmits a predetermined response signal in response to the click operation. The image display apparatus 1 receives this result and grasps that the mouse cursor has reached the response area 211.

FIG. 9B shows a value (effective movement amount LA) obtained by the mobile terminal device 2 receiving the movement amount “Lx = 290” and multiplying the multiplication rate A applied at the movement speed “Vx = 300”. ) Shows a state where the mouse cursor has just moved to the left edge of the screen of the display unit 21 as a result of moving the mouse cursor. On the other hand, the image display device 1 acquires in advance the resolution information (number of pixels 580 × 350) of the display unit 21 of the mobile terminal device 2 via the HDMI cable 3 and the wired communication unit 14. Therefore, the calibration unit 121 transmits the movement amount “Lx = 290” and the movement speed “Vx = 300”, and as a result, the mobile terminal device 2 receives the fact that the mouse cursor has reached the left end of the screen of the display unit 21. The multiplication rate A applied to “Vx = 300” can be specified as “2” (580 ÷ 290).
As described above, when the calibration unit 121 receives a response signal in the course of the “first identification process”, the calibration unit 121 identifies the multiplication rate A based on the movement amount L at the time when the response signal is received.

Note that the specific content of the first specific process is not limited to the process described above. In the example described above, the calibration unit 121 increases the movement amount Lx by 1 from the beginning. However, in other embodiments, the configuration is not limited to this mode. For example, the calibration unit 121 first increases the movement amount Lx by 100 to determine the movement amount Lx that reaches the left end of the screen, and gradually reduces the movement amount L from the stage of reaching the left end of the screen to narrow down the amount. May be. In this way, the number of repetitions required for the first specific process performed by the calibration unit 121 is reduced, and as a result, the speed of the entire calibration process can be improved. In addition, the value that the calibrating unit 121 sets as the maximum moving speed may be determined by hitting a value that is recognized to be sufficiently large in consideration of general specifications of the mobile terminal device 2.
The calibration unit 121 transmits the cursor movement instruction signal while decreasing the movement amount Lx in order from a sufficiently large value, and can receive the response signal first from the state where the response signal from the mobile terminal device 2 can be received. The multiplication rate A may be specified with reference to the movement amount Lx at the time of disappearance.

FIG. 10 is a first diagram illustrating a process of the calibration unit in the embodiment of the present invention.
The calibration unit 121 identifies that the multiplication rate A corresponding to the maximum movement speed “Vx = 300” is “2” as a result of the first identification process for the first time described above (FIG. 10).

FIG. 11 is a diagram illustrating a second specifying process of the calibration unit according to the embodiment of the present invention.
Subsequently, the calibration unit 121 obtains a range (lower limit) of the moving speed Vx to which “2” is applied for the obtained multiplication rate A. The calibration unit 121 performs a “second specifying process” for specifying the range of the moving speed to which the multiplication rate A specified in the first specifying process is applied.
Specifically, the calibration unit 121 moves the cursor while decreasing the moving speed Vx by a predetermined amount in a state where the moving amount Lx is fixed to the moving amount when reaching the left end of the screen in the first specific process. The transmission of the instruction signal and the click instruction signal is repeated a plurality of times.

First, in FIG. 11A, the calibration unit 121 transmits a cursor movement instruction signal with a movement amount “Lx = 290” and a movement speed “Vx = 299”. Here, the calibration unit 121 decrements the moving speed Vx by 1 from the value 300 fixed in the first specific process to “Vx = 299”. The mobile terminal device 2 receives the cursor movement instruction signal and moves the mouse cursor based on the received contents (movement amount “Lx = 290”, movement speed “Vx = 299”).

The calibration unit 121 then transmits a click instruction signal. Here, since the mouse cursor in the portable terminal device 2 belongs to the response area 211 as shown in FIG. 11A, the portable terminal device 2 transmits a predetermined response signal in response to the click instruction signal. Even if the moving speed is reduced by 1 from 300 to “Vx = 299”, the image display device 1 recognizes that “2” is still applied to the multiplication rate A.

Next, the calibration unit 121 once returns the mouse cursor to the reference position, and further reduces the moving speed Vx from the previous time while keeping the moving amount Lx fixed at “290”, and again moves the cursor as “Vx = 298”. An instruction signal is transmitted. The mobile terminal device 2 receives the cursor movement instruction signal and moves the mouse cursor. And if the calibration part 121 receives the response signal which should respond to the click instruction | indication signal transmitted subsequently, it will repeat the same process, reducing the moving speed Vx further.

FIG. 11B shows a state after the calibration unit 121 repeats the process of transmitting the cursor movement instruction signal while decreasing the moving speed Vx a plurality of times. In FIG. 11B, the calibration unit 121 transmits a cursor movement instruction signal with the movement amount “Lx = 290” and the movement speed “Vx = 249”. The mobile terminal device 2 receives the cursor movement instruction signal and moves the mouse cursor based on the received contents (movement amount “Lx = 290”, movement speed “Vx = 249”).

The calibration unit 121 then transmits a click instruction signal. Here, since the mouse cursor in the mobile terminal device 2 departs from the response area 211 and belongs to the non-response area 210 as shown in FIG. 11B, the mobile terminal apparatus 2 has a predetermined response to the click instruction signal. Do not send a response signal. The image display apparatus 1 receives this result and grasps that the mouse cursor has left the response area 211. That is, this result indicates that “2” is not applied to the multiplication rate A for the movement amount “Lx = 290” only after the movement speed Vx is reduced from “300” to “249”.

FIG. 12 is a second diagram showing the progress of the processing of the calibration unit in the embodiment of the present invention.
The calibration unit 121 receives the fact that the moving speed Vx at which “2” is no longer applied as the multiplication rate A is “249” as a result of the first specific processing described above, and as the multiplication rate A, It is specified that the lower limit of the moving speed Vx to which “2” is applied is “250” immediately before “249” (FIG. 12).
In the processing described above, the calibration unit 121 detects that the response signal from the mobile terminal device 2 is not received while gradually reducing the moving speed Vx, and applies “2” to the multiplication rate A. The lower limit of the moving speed Vx is specified. However, the above processing in the calibration unit 121 of the present embodiment is not limited to this mode. That is, the calibration unit 121 can first receive the response signal from a state where it cannot receive the response signal from the mobile terminal device 2 by transmitting the cursor movement instruction signal while increasing the moving speed Vx sequentially from “1”, for example. The lower limit of the moving speed Vx at which “2” is applied to the multiplication rate A may be specified with reference to the moving speed Vx at the time of transition to the state.

FIG. 13 is a second diagram illustrating the first specifying process of the calibration unit according to the embodiment of the present invention.
Subsequently, the calibration unit 121 obtains a multiplication rate A applied to the next moving speed Vx (moving speed Vx of 249). The calibration unit 121 first performs the second “first identification process” for identifying the multiplication rate A corresponding to the moving speed “Vx = 249”.

When the moving speed Vx is set to “249” in the second specifying process, “2” is not applied to the multiplication rate A, and the calibrating unit 121 moves the mouse cursor to the response area 211 (the left end of the screen of the display unit 21). ) Is no longer reached. Further, the calibration unit 121 knows that the movement amount Lx when the mouse cursor stops reaching the response area 211 is “290”. Therefore, the calibration unit 121 repeats the transmission of the cursor movement instruction signal and the click instruction signal a plurality of times while increasing the movement amount Lx by a predetermined amount from “290” in a state where the movement speed Vx is fixed at “249”.

First, in FIG. 13A, the calibration unit 121 transmits a cursor movement instruction signal with a movement amount “Lx = 291” and a movement speed “Vx = 249”. The mobile terminal device 2 receives the cursor movement instruction signal and moves the mouse cursor based on the received contents (movement amount “Lx = 291”, movement speed “Vx = 249”). At this time, the image display device 1 cannot grasp how much the position of the mouse cursor in the mobile terminal device 2 has actually moved from the reference position.

The calibration unit 121 then transmits a click instruction signal. When the mobile terminal device 2 receives the click instruction signal, the mobile terminal device 2 executes a process when a click operation using a mouse is received. Here, since the mouse cursor in the mobile terminal device 2 belongs to the non-response area 210 as shown in FIG. 13A, the mobile terminal device 2 does not transmit a predetermined response signal in response to the click operation. The image display device 1 receives this result and grasps that the mouse cursor has not reached the response area 211 (that is, the end of the display unit 21).

Next, after returning the mouse cursor to the reference position, the calibration unit 121 increases the moving amount Lx from the previous time while keeping the moving speed Vx fixed at “249”, and sets the cursor moving instruction signal again as “Lx = 292”. Send. The mobile terminal device 2 receives the cursor movement instruction signal and moves the mouse cursor. When the calibration unit 121 determines that there is no response signal to be responded to the click instruction signal transmitted subsequently, the calibration unit 121 repeats the same processing while further increasing the movement amount Lx.

FIG. 13B shows a state after the calibration unit 121 repeats the process of transmitting the cursor movement instruction signal while increasing the movement amount Lx a plurality of times. In FIG. 13B, the calibration unit 121 transmits a cursor movement instruction signal with a movement amount “Lx = 310” and a movement speed “Vx = 249”. The mobile terminal device 2 receives the cursor movement instruction signal and moves the mouse cursor based on the received contents (movement amount “Lx = 310”, movement speed “Vx = 249”).

The calibration unit 121 then transmits a click instruction signal. Here, since the mouse cursor in the mobile terminal device 2 has reached the response area 211 as shown in FIG. 13B, the mobile terminal device 2 transmits a predetermined response signal in response to the click instruction signal. The image display apparatus 1 receives this result and grasps that the mouse cursor has reached the response area 211.

When the calibration unit 121 transmits the movement amount “Lx = 310” and the mouse cursor has reached the left end of the screen of the display unit 21, the multiplication rate A for the movement speed “Vx = 249” is “1.87”. It can be specified that (580 ÷ 310).

FIG. 14 is a third diagram illustrating the progress of the processing of the calibration unit according to the embodiment of the present invention.
The calibration unit 121 identifies that the multiplication rate A corresponding to the moving speed “Vx = 249” is “1.87” as a result of the second “first identification process” (FIG. 14). Subsequently, the calibration unit 121 executes the second “second specific process” and obtains the range (lower limit) of the moving speed Vx to which “1.87” is applied for the multiplication rate A.

Thereafter, the calibration unit 121 repeatedly executes the contents of the “first specifying process” and the “second specifying process” described with reference to FIGS. 9 to 14, and the movement amount set in the mobile terminal device 2. All the contents of the multiplication rate information table are specified. Note that all the above explanations are processing procedures for specifying the movement amount multiplication rate information in the X-axis direction, but the calibration unit 121 specifies the movement amount multiplication rate information in the Y-axis direction according to the same processing procedure. To do.

15 and 16 are a first diagram and a second diagram illustrating a processing flow of the calibration unit and the mobile terminal device according to the embodiment of the present invention.
The processing flow of the calibration unit 121 of the image display device 1 and the mobile terminal device 2 described above will be described in order with reference to FIGS. 15 and 16. 15 and 16 show a processing flow after the mobile terminal device 2 is connected to the image display device 1 by wire.

First, when the image display device 1 receives a predetermined connection signal from the portable terminal device 2 via the wired communication unit 14, the image display device 1 establishes wired communication and wireless communication with the portable terminal device 2 (step S10). At this stage, the image display device 1 acquires information related to the mobile terminal device 2, particularly the aspect ratio of the display unit 21, resolution information, and the like. Next, the calibration unit 121 transmits a predetermined calibration application activation instruction signal to the mobile terminal device 2 via the wireless communication unit 13 (step S11). When receiving the calibration application activation instruction signal, the control unit 22 of the portable terminal device 2 reads and activates the calibration application program (step S12). When the activation of the calibration application function is completed, the control unit 22 of the portable terminal device 2 notifies that (step S13).

The proofreading unit 121 starts the calibration process upon receiving the start completion notification of the calibration application function. First, the calibration unit 121 starts the “first identification process” (step S14). The calibration unit 121 transmits a cursor movement instruction signal and a click instruction signal set to predetermined values (step S15). Here, the calibration unit 121 sets the movement speed V of the cursor movement instruction signal to the maximum movement speed (for example, V = 300).

When receiving the cursor movement instruction signal, the mobile terminal device 2 moves the mouse cursor based on the cursor movement instruction signal, and subsequently executes a mouse click operation based on the received click instruction signal (step S16). . However, when the position of the moved mouse cursor belongs to the non-response area 210, the mobile terminal device 2 does not transmit a predetermined response signal (step S17 (state shown in FIG. 9A)). Note that the portable terminal device 2 may separately transmit a signal notifying that the position of the mouse cursor belongs to the non-response area 210 in step S17.

When the calibration unit 121 detects that there is no response signal from the mobile terminal device 2 for a predetermined period after transmitting the click instruction signal, the calibration unit 121 transmits a cursor movement instruction signal for returning the mouse cursor in the mobile terminal device 2 to the reference position ( Step S18). When receiving the cursor movement instruction signal transmitted in step S18, the portable terminal device 2 moves the mouse cursor to the reference position based on the cursor movement instruction signal (step S19). On the other hand, the calibration unit 121 receives the absence of a predetermined response signal from the mobile terminal device 2 and increases the movement amount L in a state where the movement speed V in the cursor movement instruction signal is fixed (step S20). Steps S15 to S20 are repeated until a predetermined response signal is received from 2.

When the mobile terminal device 2 receives the click instruction signal when the mouse cursor reaches the response area 211, the mobile terminal device 2 notifies the response signal (step S21 (state shown in FIG. 9B)). Upon receiving the response signal, the calibration unit 121 identifies the multiplication rate A in the movement amount multiplication rate information table determined by the mobile terminal device 2 and ends the first identification process (step S22).

Subsequently, the calibration unit 121 of the image display device 1 starts the “second specifying process” (step S23 in FIG. 16). In the second specifying process, the calibration unit 121 first transmits a cursor movement instruction signal and a click instruction signal based on the first specifying process (step S24). The calibration unit 121 sets the movement amount L of the cursor movement instruction signal to a value when the response signal is first received in the first specific process. In addition, the calibration unit 121 sets the moving speed V to be lower than the value when the response signal is first received in the first specific process.

When receiving the cursor movement instruction signal, the mobile terminal device 2 moves the mouse cursor based on the cursor movement instruction signal, and subsequently executes a mouse click operation based on the received click instruction signal (step S25). . However, even if the moving speed V is decreased, if the position of the moved mouse cursor still belongs to the response area 211, the mobile terminal device 2 transmits a predetermined response signal (step S26 (shown in FIG. 11A)). Status)).

When the calibration unit 121 detects that there is a response signal from the mobile terminal device 2, the calibration unit 121 transmits a cursor movement instruction signal for returning the mouse cursor in the mobile terminal device 2 to the reference position (step S27). When receiving the cursor movement instruction signal transmitted in step S27, the portable terminal device 2 moves the mouse cursor to the reference position based on the cursor movement instruction signal (step S28). On the other hand, the calibration unit 121 receives the response signal from the mobile terminal device 2 and decreases the moving speed V in a state where the movement amount L in the cursor movement instruction signal is fixed (step S29). Steps S24 to S29 are repeated until no predetermined response signal is received.

When the mobile terminal device 2 receives the click instruction signal when the mouse cursor is out of the response area 211, the mobile terminal device 2 does not notify the response signal (step S30 (state shown in FIG. 11B)). The calibration unit 121 detects that the response signal has not been received, specifies the lower limit of the moving speed V to which the multiplication rate A specified in the first specifying process is applied, and performs the second specifying process. The process ends (step S31). When the first “first specific process” and the “second specific process” are completed, the calibration unit 121 sets the next stage multiplication rate A and the range of the moving speed V to which the multiplication rate A is applied. In order to specify, the second “first specifying process” is started (step S32).

FIG. 17 is a diagram for specifically explaining the operation of the cursor operation information generation unit according to the embodiment of the present invention.
Next, an example of processing performed by the cursor operation information generation unit 120 based on the movement amount multiplication rate information specified by the calibration unit 121 in the above-described processing will be described. First, it is assumed that the resolution in the display unit 21 of the mobile terminal device 2 is 600 × 400 and the resolution in the display unit 11 of the image display device 1 according to the aspect ratio of the display unit 21 is 900 × 600 (FIG. 17). . It is assumed that the calibration unit 121 has already completed the calibration process.

Here, it is assumed that the touch panel 10 detects a touch operation at the coordinates (300, 0) on the display unit 11 (right side in FIG. 17). Then, the cursor operation information generation unit 120 generates a cursor movement instruction signal for clicking on a position corresponding to the touch detection location on the display unit 21 of the mobile terminal device 2. Specifically, the cursor operation information generation unit 120 considers the resolution ratio P (600/900 = 2/3) which is the scale of the display unit 11 and the display unit 21, and coordinates (300, The mouse cursor may be moved from the reference position (0, 0) by a value (200, 0) obtained by multiplying 0) by the resolution ratio P (2/3).

Therefore, the cursor operation information generation unit 120 further determines the movement amount Lx in consideration of the multiplication rate A to the movement amount Lx corresponding to the movement speed Vx. That is, the cursor operation information generation unit 120 needs to immediately reflect the touch operation on the self-image display device on the mobile terminal device 2, so the mouse cursor on the mobile terminal device 2 moves to the coordinates (200, 0) at the fastest speed. A cursor movement instruction signal is generated so that the That is, the cursor operation information generation unit 120 sets the movement speed Vx as the maximum movement speed (for example, “300”).

Then, the cursor operation information generation unit 120 refers to the movement amount multiplication rate information table acquired by the calibration unit 121 and acquires the multiplication rate A (for example, “2”) corresponding to the maximum movement speed. Then, the cursor operation information generation unit 120 sets the movement amount Lx to “100” in consideration of the multiplication rate “A = 2” so that the mouse cursor is moved by 200 (so that the effective movement amount LA becomes 200). To do.

As described above, the contents of the cursor movement instruction signal generated by the cursor operation information generation unit 120 are the movement amount “Lx = 100” and the movement speed “Vx = 300”. When receiving the cursor movement instruction signal, the portable terminal device 2 moves the mouse cursor from the reference position (0, 0) based on the contents thereof. The mobile terminal device 2 multiplies the movement amount “Lx = 100” by the multiplication rate “2” based on the movement speed “Vx = 300” and moves the mouse cursor to the X axis based on the effective movement amount “LAx = 200”. Move 200 in the direction (left in FIG. 17).

That is, the cursor operation information generation unit 120 determines the amount of movement of the cursor movement instruction signal to be generated based on Expression (1).

In the formula (1), the touch position (X) is a touch position (X coordinate value) detected by the touch panel 10 of the image display device 1. In the example of FIG. 17, X = 300. Further, the resolution ratio (P) between the two display units is a resolution ratio between an image displayed on the display unit 11 of the image display device 1 and an image displayed on the display unit 21 of the mobile terminal device 2. In the example of FIG. 17, the resolution in the display unit 21 of the mobile terminal device 2 is 600 × 400, and the resolution in the display unit 11 of the image display device 1 according to the aspect ratio of the display unit 21 is 900 × 600. P = 2/3. The applied multiplication rate (A) is a multiplication rate A corresponding to the moving speed Vx to be transmitted by the cursor operation information generation unit 120. The cursor operation information generation unit 120 acquires the multiplication rate A with reference to the movement amount multiplication rate information table acquired in advance by the calibration unit 121. In the example shown in FIG. 17, since the moving speed Vx is “300” which is the maximum moving speed, “2” is applied to the multiplication rate A.

The cursor operation information generation unit 120 applies the above value to Expression (1) to obtain the movement amount Lx. In the example illustrated in FIG. 17, the cursor operation information generation unit 120 obtains “100” as the movement amount Lx.

Even when the user performs a drag operation on the image display device 1, the cursor operation information generation unit 120 performs the same processing as in the case of the touch operation by referring to the movement amount multiplying rate information table. be able to. In other words, the cursor operation information generation unit 120 sets various movement speeds Vx according to the speed of the finger that the user slides in the drag operation. Each time, the multiplication applied from the movement amount multiplication rate information table is performed. What is necessary is just to acquire the rate A and apply it to Formula (1).

In the image display device of the present embodiment described above, it becomes possible to operate the mobile terminal device 2 from the image display device 1 without requiring a dedicated application program for remote operation from the image display device 1, and The effect of improving operability can be obtained. Although the image display device 1 needs to install a calibration application program in the portable terminal device 2 for the calibration process, the calibration application program does not operate after the calibration process is completed. The operability is not affected.

Also, the image display device 1 includes a calibration unit 121, and the image display device 1 performs automatic calibration processing. In this way, the image display device 1 can reflect the operation intended by the user on the mobile terminal device 2 regardless of the model of the mobile terminal device 2 to be linked, the user setting, the difference in mouse acceleration, and the like.

In addition, the image display apparatus 1 in this embodiment is not limited to the aspect used as a car navigation system mentioned above. For example, the image display device 1 is a large-screen television having a display unit larger than that of the mobile terminal device 2, a display for a personal computer, or a tablet-type terminal device, and a predetermined information processing system ( In addition to a car navigation system, it may function as a video player, game machine, or the like. On the other hand, although the portable terminal device 2 was demonstrated as a general smart phone, the portable terminal device 2 in this embodiment may be a tablet-type terminal device of a size larger than a smart phone, for example.

The image display device 1 and the mobile terminal device 2 described above have a computer system inside. The processes of the image display device 1 and the portable terminal device 2 described above are stored in a computer-readable recording medium in the form of a program, and the program is read and executed by the computer. Is called. Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM (Compact Disk Only Memory), a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in embodiment mentioned above. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2012-279622 filed on Dec. 21, 2012, the entire disclosure of which is incorporated herein.

The present invention can be used, for example, for remote operation of a mobile terminal device such as a smartphone. By applying the present invention, it becomes possible to remotely operate the portable terminal device without requiring a dedicated application program for remote operation, and the operability is improved.

DESCRIPTION OF SYMBOLS 1 ...

Image display apparatus

10, 20 ... Touch panel 100 ...

Touch detection part

11, 21 ... Display part 210 ... Non-response area 211 ...

Response area

12, 22 ... Control part 120 ... Cursor operation information generation unit 121 ...

Calibration unit

13 and 23 ... Wireless communication unit 130 ... Cursor operation

information transmission unit

14 and 24 ... Wired communication unit 3 ... HDMI cable

Claims

An image display unit for displaying an image displayed on the display screen of the mobile terminal device;
A touch operation detection unit that detects a touch operation on the display screen of the self-image display device;
Based on the touch operation, a cursor operation information generation unit that generates a cursor movement instruction signal and a click instruction signal conforming to a predetermined communication method;
A cursor operation information transmission unit for transmitting the cursor movement instruction signal and the click instruction signal according to the predetermined communication method;
The position designated by the touch operation on the display screen of the self-image display device and the position of the cursor moved based on the cursor movement instruction signal on the display screen of the mobile terminal device, respectively. And a calibration unit for obtaining calibration information for relatively setting the same position with reference to the image displayed by the mobile terminal device;
An image display device comprising:
The calibration unit is
Via the cursor operation information transmission unit, the transmission of the cursor movement instruction signal and the transmission of the click instruction signal following the transmission of the cursor movement instruction signal are repeated a plurality of times, and the mobile terminal device responds to the click instruction signal from the portable terminal device. The image display device according to claim 1, wherein the calibration information is acquired based on presence or absence of a predetermined response signal.
The cursor movement instruction signal is
It is a combination of two types of information: movement amount and movement speed.
The calibration unit is
Acquiring the calibration information by specifying movement amount multiplying rate information that is specific information determined in the mobile terminal device and that determines a multiplying rate to be applied to the moving amount according to the moving speed The image display device according to claim 2.
The calibration unit is
First, the transmission of the cursor movement instruction signal and the transmission of the click instruction signal following the transmission of the cursor movement instruction signal are repeated a plurality of times while changing the movement amount by a predetermined amount with the movement speed fixed. Specific processing of
The transmission of the cursor movement instruction signal and the transmission of the click instruction signal following the transmission of the cursor movement instruction signal are repeated a plurality of times while changing the movement speed by a predetermined amount with the movement amount fixed. The image display device according to claim 3, wherein the specific processing is performed a plurality of times.
The calibration unit is
When the response signal is received in the course of the first specific process, the multiplication rate is specified based on the movement amount at the time when the response signal is received, and the second specific process is started.
If the response signal is no longer received in the course of the second specific process, the multiplication rate specified in the first specific process is based on the moving speed at the time when the response signal is no longer received. The image display apparatus according to claim 4, wherein a range of a moving speed to which is applied is specified.
The cursor operation information generation unit
The movement amount is divided into divided movement amounts to be transmitted per transmission, and the cursor movement instruction signal is generated using the divided movement amount as a movement speed,
The cursor operation information transmission unit
The image display according to any one of claims 3 to 5, wherein the information on the movement amount and the movement speed is transmitted by transmitting the divided movement amount in a plurality of times for each transmission. apparatus.
An image display device that displays an image displayed on the display screen of the mobile terminal device, and a mobile terminal device that performs a predetermined calibration process,
A control unit that receives a cursor movement instruction signal compliant with a predetermined communication method and moves the cursor on the display screen of the portable terminal device based on the cursor movement instruction signal;
The control unit further includes:
When a predetermined response area is set on a part of the display screen of the portable terminal device, and the cursor is in the response area, when a predetermined click instruction signal is received from the image display apparatus A portable terminal device that transmits a predetermined response signal.
An image display device for displaying an image displayed on the display screen of the mobile terminal device;
The image display device, a portable terminal device for performing a predetermined calibration process,
An information processing system comprising:
The image display device includes:
A touch operation detection unit that detects a touch operation on the display screen of the self-image display device;
Based on the touch operation, a cursor operation information generation unit that generates a cursor movement instruction signal and a click instruction signal conforming to a predetermined communication method;
A cursor operation information transmission unit for transmitting the cursor movement instruction signal and the click instruction signal according to the predetermined communication method;
The position specified by the touch operation on the display screen of the self-image display device and the position of the cursor moved based on the cursor movement instruction signal on the display screen of the mobile terminal device are displayed on the image, respectively. A calibration unit for acquiring calibration information for relatively setting the same position with reference to an image displayed by the device and the mobile terminal device;
With
The calibration unit further includes
Via the cursor operation information transmission unit, the transmission of the cursor movement instruction signal and the transmission of the click instruction signal following the transmission of the cursor movement instruction signal are repeated a plurality of times, and the mobile terminal device responds to the click instruction signal from the portable terminal device. Based on the presence or absence of a predetermined response signal, obtain the calibration information,
The portable terminal device
A control unit that receives the cursor movement instruction signal and moves the cursor on the display screen of the mobile terminal device based on the cursor movement instruction signal;
The control unit further includes:
A predetermined response area is set on a part of the display screen of the mobile terminal device, and the response signal is transmitted when the click instruction signal is received when the cursor is in the response area Information processing system.
Display the image displayed on the display screen of the mobile terminal device on another display screen,
The position touched on the other display screen displaying the image and the position of the cursor moved based on a predetermined cursor movement instruction signal on the display screen of the mobile terminal device, respectively, Obtaining calibration information for making the position relatively the same with reference to the image displayed on the display screen and the image displayed on the display screen of the mobile terminal device;
Detecting a touch operation on the other display screen;
Based on the touch operation, the cursor movement instruction signal and the click instruction signal are generated,
An image display apparatus control method for transmitting the cursor movement instruction signal and the click instruction signal according to a predetermined communication method.
A computer of an image display device that displays an image displayed on the display screen of the mobile terminal device,
A touch operation detection unit for detecting a touch operation on the display screen of the self-image display device;
A cursor operation information generation unit that generates a cursor movement instruction signal and a click instruction signal based on a predetermined communication method based on the touch operation,
A cursor operation information transmitting unit for transmitting the cursor movement instruction signal and the click instruction signal according to the predetermined communication method;
The position designated by the touch operation on the display screen of the self-image display device and the position of the cursor moved based on the cursor movement instruction signal on the display screen of the mobile terminal device, respectively. And a calibration unit that acquires calibration information for relatively setting the same position with reference to the image displayed by the mobile terminal device,
Program to function as.