WO2017136961A1

WO2017136961A1 - Improved remote screen control

Info

Publication number: WO2017136961A1
Application number: PCT/CN2016/073738
Authority: WO
Inventors: Huajie WU; Kuichu NI
Original assignee: Qualcomm Technologies International, Ltd.
Priority date: 2016-02-11
Filing date: 2016-02-11
Publication date: 2017-08-17

Abstract

The present embodiments implement Bluetooth® HID to locally back- control a smartphone that has been linked to a vehicle's hands free device, and then implementing the Bluetooth® Serial Port Profile (SPP) to aid the Bluetooth® HID in the back-controlling of the smartphone in the car kit locally. In particular, the present embodiments address the issue of floating pointer commonly experienced when back-controlling a host HID by implementing a method of sending a HID input report periodically according to the dynamic velocity of the remote pointer's movement, that is, the movement of the smartphone cursor, and re-sampling the touch message captured by the car kit.

Description

IMPROVED REMOTE SCREEN CONTROL

TECHNICAL FIELD

The present invention relates to communication devices and associated methods. In particular， the present invention relates to a method and system for remote back-control of communication devices.

BACKGROUND OF RELATED ART

Using a mobile phone device or other handheld computing devices whilst operating a moving vehicle is both impractical and dangerous， and so it is common to link such devices to an on-board hands-free device installed within the vehicle. The on-board hands-free device may be integrated into the vehicles hardware， or it may be an external device typically mounted to the dashboard of the vehicle. Generally， smartphones can cast the screen into the car kit through communication protocols such as HDMI (High Definition Multimedia Interface) ， MHL (Mobile High-Definition Link) or Miracast. However， HDMI and MHL do not enable the in car kit to back-control the smartphone， and whilst Miracast does define a user input back channel (UIBC) for user input back control， it is not supported by many smartphone devices available on the market.

Once the smartphone has cast the screen into the car kit via the established communication protocol， the car kit may use an input device such as a touchscreen or a

Human Interface Device (HID) ， for example， a mouse， trackball， touchpad or any other suitable device to back-control the smartphone.

When using the touchscreen or

HID， the car kit receives a touch message which prompts the car kit to generate a HID input report which is sent to the smartphone via the established communication protocol. However， if the user writes quickly using the touch screen or

HID， the user’s input cannot always be resolved effectively enough by the smartphone， and as a result causes the pointer on the screen of the smartphone to float. Consequently， the output on the smartphone screen becomes unintelligible. In most smartphones， for example， phones based on the Android operating system (OS) ， the pointer sensitivity is measured based on two characteristics； the speed of the pointer and the acceleration of the pointer， which will be discussed in more detail below. If the HID input report for the touch screen or

HID is sent out too quickly， such that the speed of the user input exceeds the default low threshold for the smartphone’s pointer speed， the smartphone is prompted to accelerate the movement of the pointer which causes the pointer to move out of control. To address this， the usual method is to reduce the touch message received by the touchscreen or

HID in order to decrease the number of HID input reports sent to the smartphone in order to then accelerate the pointer’s response time in the smartphone. However， this does not consider the speed at which the pointer is being caused to move， and as a result the writing being produced by the pointer may still be unintelligible.

SUMMARY

This Summary is provided to introduce in a simplified form a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter， nor is it intended to limit the scope of the claimed subject matter.

The present invention addresses the above noted issue of floating pointer commonly experienced when back-controlling a host HID by implementing a method of sending a HID input report periodically according to the dynamic velocity of the remote pointer’s movement， that is， the movement of the smartphone cursor， and re-sampling the touch message captured by the car kit to ensure that the HID input report is sent from the car kit and received by the smartphone on a periodic basis

In a first aspect， the present invention provides a method of transmitting data between a first device and a second device， the method comprising receiving user input data at the first device and transmitting the user input data at a first transmission rate to the second device via a first communication protocol， determining the speed at which the user input data is received by the first device and comparing it with the speed at which the second device is configured to receive the user input data， re-sampling the user input data if the speed at which the user input data is received by the first device is higher than the speed at which the second device is configured to receive the user input data， determining a second transmission rate based on the re-sampled rate， and transmitting the user input data from the first device to the second device at the second transmission rate.

As such， if the first device is receiving user input data faster than the second device is configured to receive user input device， it will adjust the transmission rate so as to match the second device’s capability. Therefore， it will re-sample the user input data at a rate corresponding to the speed at which the second device is configured to receive user input data， decreasing the transmission rate so as to transmit data at a rate that the second device is able to handle. Consequently， the second device is able to clearly output the received user input data without error.

In some embodiments， the first communication protocol is a

Human-Interface Device (HID) profile.

The method may further comprise transmitting user data between the first device and the second device via a second communication protocol， wherein the second communication protocol may be the

Serial Port Profile (SPP) . As such， the second communication protocol is such that communication between the first device and the second device includes retrieving information from the second device relating to the speed at which the second device is configured to receive the user input data and transmitting it to the first device.

In some embodiments， the first transmission rate is based on the rate at which user input data is received at the first device. In this respect， all user input data will be transmitted to the second device as soon as it is received.

Preferably， the second transmission rate is lower than the first transmission rate， the second transmission rate being such that the user input data is periodically sent from the first device to the second device. In this respect， user input data will not necessarily be sent as soon as it is received. For example， the second transmission rate may be such that user input data is transmitted for every third unit of user input data received at the first device.

In further embodiments， the re-sampled rate is determined in dependence on the speed at which the second device is configured to receive the user input data. That is to say， the first device will re-sample the user input data at a rate corresponding to the speed at which the second device is configured to receive user input data. For example， the speed at which the second device is configured to receive the user input data may be at least 500 pixels/sec.

In a second aspect， the present invention provides a device for communicating with a further device， the device comprising a receiver for receiving input data， a transmitter for transmitting the user input data at a first transmission rate to the further device via a first communication protocol， a processor configured to determine the speed at which the user input data is received by the device and to compare it with the speed at which the further device is configured to receive the user input data， the processor is further configured to re-sample the user input data received by the device if the speed at which the user input device is received by the device is higher than the speed at which the second device is configured to receive the user input data， and to determine a second transmission rate based on the re-sampled rate， and wherein the transmitter is arranged to transmit the user input data to the second device at the second transmission rate.

In some embodiment， the device may be an in-vehicle communication device.

The device may further comprise a touch screen， a Human Interface Device (HID) mouse and/or a HID consumer control.

The first communication protocol may be a

Human-Interface Device (HID) profile.

In other embodiments， the device is further arranged to communicate with the further device via a second communication protocol， wherein the second communication protocol may be the

Serial Port Profile (SPP) . As such， the second communication protocol is such that communication between the device and the further device includes retrieving information from the further device relating to the speed at which the further device is configured to receive the user input data and transmitting it to the device.

In embodiments of the present invention， the first transmission rate is based on the rate at which user input data is received， the second transmission rate being lower than the first transmission rate， and the second transmission rate being such that the user input data is periodically sent from the device to the further device.

The re-sampled rate is determined in dependence on the speed at which the further device is configured to receive the user input data

BRIEF DESCRIPTION OF THE DRAWINGS

The example embodiments are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings. Like numbers reference like elements throughout the drawings and specification.

Figure 1 shows example apparatus according to embodiments of the present invention.

Figure 2 illustrates example system architecture according to embodiments of the present invention；

Figure 3 illustrates a sampling method according to embodiments of the present invention；

Figure 4 is a flow diagram illustrating an embodiment of the present invention；

Figure 5 is a sequence diagram according to embodiments of the present invention.

DETAILED DESCRIPTION

The example embodiments are described below in the context of ranging operations performed by and between Wi-Fi enabled devices for simplicity only. It is to be understood that the example embodiments are equally applicable for performing ranging operations using signals of other various wireless standards or protocols， and for performing ranging operations between various devices (e.g.， between a STA and a wireless AP， between APs， between STAs， and so on) . Thus， although the example embodiments are described below in the context of a WLAN system， the example embodiments are equally applicable to other wireless networks (e.g.， cellular networks， pico networks， femto networks， satellite networks) ， as well as for systems using signals of one or more wired standards or protocols (e.g.， Ethernet and/or HomePlug/PLC standards) . As used herein， the terms WLAN and Wi-Fi may include communications governed by the IEEE 802.11 standards， Bluetooth， HiperLAN (a set of wireless standards， comparable to the IEEE 802.11 standards， used primarily in Europe) ， and other technologies having relatively short radio propagation range. Thus， the terms “WLAN” and “Wi-Fi” may be used interchangeably herein.

Figure 1 shows an on-board hands-free smartphone link system 1 according to embodiments of the present invention， which comprises a first device 10 such as smartphone or other suitable handheld device and a second device 11， for example， an integrated car kit having an input/output interface 13 arranged to receive control inputs and provide output information via a plurality of input and output hardware (13a-e) located within a vehicle (not shown) . In one example， the second devicet 11 receives user input from a

HID provided on the second devicet 11 or on another part of the vehicle’s interior such as， for example， control buttons on the steering wheel 13a in order to seamlessly back-control the first device 10. The second devicet 11 may also receive user input from a touch screen device 13b or an audio input device 13e， for example， a microphone. It will be appreciated that any suitable means of entering user input data on the second device 11 may be employed. For this reason， details of entering data on the second device will not be further described. Information may then be output， for example， to a display screen 13c， via an audio output device 13d， for example， speakers.

The second device 11 further comprises a processor unit 14， a working memory 15 and a computer readable storage medium 16 upon which various programs are stored and arranged to control the second device 11， as will be described in more detail below.

The first device 10 and the second device 11 communicate with each other via a communication protocol 17， for example， HDMI， MHL or Miracast. Through this communication protocol 17， the screen 12 of the first device 10 is cast into the display screen 13c of the second device 11.

As such， it is important that the first device 10 is capable of supporting the

HID host， wherein the

HID host may be any input device connected to the second device 11 and suitable for back controlling the screen 12 of the first device 10， for example， a HID mouse， HID touch screen or HID consumer control.

A HID mouse is a hand-held， button activated input device that when rolled along a flat surface， directs an indicator to move correspondingly about a visual display screen， allowing the user to move the indicator freely in select operations or to manipulate the text or graphics output to the visual display screen. A mouse typically consists of two axes (X and Y) and one， two or three buttons.

A HID touch screen is a digitizer with an integrated display that allows the use of a finger or stylus for pointing. Some touch screen technologies can differentiate between the touch of a finger and the touch of a stylus. Typically， the HID touch screen supports the absolute X and Y axis， instead of the relative X and Y axis.

HID consumer control is a general consumer control device. Typically， there are some additional virtual keys in the smartphone which aren’t in the scope of the remote screen. For example， Android phones normally have 3 additional virtual keys， including ‘Back’ ， ‘Home’a nd ‘Menu’ . As a HID mouse can’t be fitted with controls such as virtual keys， HID consumer control can provide such virtual keys.

Figure 2 shows the system architecture 2 used within system 1 of FIG. 1. The system architecture 2 comprises a

Application 20 in communication with a Smartphone Link Application 23 via an Inter-Process Communication (IPC) . The IPC between

Application 20 and Smartphone Link Application 23 depends on the Operating System (OS) being used. For example， Windows Message Queue or Named Event can be utilized for IPC in Microsoft Windows CE 6.0. The

Application 20 communicates with Smartphone Link Application 23 to exchange information regarding the connection status of the

HID and the communication protocol 17， that is， the communication protocol 17 connecting the first device 10 with the second device 11. Typically， the

Application 20 is run in user space under different OS， such as WinCE6.0 or Linux. The Smartphone Link Application 23 is arranged to cast the smartphone’s screen 12 into a suitable system on chip (SoC) platform， through a communication protocol 17 such as HDMI， MHL or Miracast. Miracast is a peer-to-peer wireless screencast standard formed via WiFi^TM direct connections which enables wireless or wired delivery of compressed standard or high-definition video to or from desktops， tablets， mobile phones， and other devices having visual displays.

The

Application 20 is connected to a suitable

stack 21， such as Cambridge Silicon Radio Synergy^TM for Windows Embedded， via an IPC. The

stack 21 communicates with a

chip 22， such as Cambridge Silicon Radio’s

chip CSR8311-A08， through Host Controller Interface (HCI) transport， for example， the Cambridge Silicon Radio

Serial Protocol. The

stack 21 is typically run in kernel space under different Operating Systems. As such， the

stack 21 is arranged to implement the protocols defined by the

Application 20.

Therefore， to control the first device 10 through

HID， the

stack 21 in the SoC platform must be able to support a HID profile， the profile being capable at the very least of registering a HID device， accepting and establishing a connection with a HID device， and disconnecting from a HID device.

The SoC platform thus acts effectively as a HID device， sending HID input reports to the first device 10 according to the touch message captured in the SoC platform via the HID mouse， HID touch screen， HID consumer control or whichever device is being used to back control the first device 10.

Notably， the first device 10 will always be able to initiate a HID connection when a

connection is established from the first device 10， provided that the HID Service Discovery Protocol (SDP) record is activated in the SoC platform. In contrast， if the SoC platform initiates a HID connection with a first device 10， and it is for the first time， the first device 10 will reject the HID connection due to the first device’s 10 native limitations. Consequently， the

Application 20 is only able to initiate HID connection with the first device 10 after a HID connection between the SoC platform and the first device 10 has been established once before.

In order to use a HID mouse， it is necessary to map the screen coordinates between local screen in the second device 11 and the remote screen in the first device 10. The HID mouse only supports the relative X and Y axis， however， the touch message captured in the SoC platform normally supports absolute X and Y in the local coordinate， which is relative with the zero point in left-upper corner of the local screen. Furthermore， it is necessary to know the previous position of the mouse cursor in the first device 10 along with the initial position of the mouse cursor. The HID mouse neither supports multi-touch features built in to the first device 10.

As indicated above， HID consumer control can provide virtual keys needed to back control the first device 10 which are not provided by a HID mouse. The

Application 20 will thus construct a HID report for a “HID combo device” comprising the HID mouse and the HID consumer control. HID consumer control can also be fitted with control of the media player in the first device 10， especially if the first device 10 is unable to cast the surface layer of media player into the SoC platform when the Smartphone Link Application 23 is active. For example， Android phones may not cast the surface layer of their media player into the SoC platform when HDMI is connected.

HID touch screen is often the better HID pointer device for controlling the first device 10， since there is HID touch screen in SoC platform by default which can support the feature of multi-touch. However， HID touch screen is not well supported well by many first devices 10. In particular， touch screen mapping between the SoC platform and the first device 10 may not be matched fully. The resolution of the local screen in the SoC platform is normally 800 x 480 pixels， whilst the resolution of the smartphone screen 12 is always different from local screen， for example， 1280 x 720 pixels. Therefore， it is impossible for the

Application 20 to pre-define the maximum value of the X axis and Y axis in HID report of the touch screen. If the

Application 20 constructs a HID report of the touch screen with local screen’s width as a maximum value of the X axis and local screen’s height as maximum value of the Y axis， the first device 10 can only make a partial region of the remote screen effective. This is due to difference between the resolution of the local screen in the SoC platform and the remote screen in the first device 10.

For the

Application 20 to control the first device 10 after the screen 12 has been cast into a SoC platform， the prerequisite is that

Application 20 should know exactly the output rectangle in the input/output interface 13 when remote screen 12 is in landscape mode or portrait mode. By default， neither the

HID (for example， a mouse or consumer control) nor the Smartphone Link Application 23 can provide exact information about the remote screen 12， such as the remote screen’s orientation and output rectangle， that is， the dimensions of the portion of the screen that displays the visual output. Therefore， using the

Serial Port Profile (SPP) is a feasible solution to address this issue， since most smartphones that are capable of supporting

HID can also support

SPP.

As such， the first device 10 needs to install a

SPP application (not shown) ， hereinafter referred to as HIDHelper. The

Application 20 cannot establish a

SPP connection until the relevant HID is connected with the first device 10. If the HID connection is lost， the

Application 20 will correspondingly disconnect from the

SPP. Whilst the

stack in the SoC platform can normally support multiple

SPP connections， the HIDHelper installed in the smartphone can usually only support one

SPP connection.

Once the

SPP connection has been made， the

Application 20 is arranged to send at least one request to the HIDHelper installed in the first device 10 asking it to retrieve information from the remote screen 12. Such requests may include information regarding the screen resolution， the screen’s direction， the screen’s dimensions or information concerning the movement of the screen’s pointer or cursor. The requests may also prompt screen calibration. The communication between

Application 20 in the SoC platform and the HIDHelper in the first device 10 through the

SPP is akin to the architecture of a Client-Server model. That is to say， a request is issued from

Application 20 which is responded to by the HIDHelper in the first device 10. The HIDHelper can also send out a voluntary indicator to the

Application 20， for example， the HIDHelper can initially inform the

Application 20 of the remote screen’s 12 direction or pointer movement speed， since such information may be updated dynamically by the first device 10.

Once the first device 10 has been connected to the second device 11 and any necessary screen calibration completed， the remote screen cast from the first device 10 to the second deivice 11 can be controlled locally via touchscreen or a

HID such as a mouse. User input via such devices is captured by the

Application 20 as a touch message.

Generally， the

Application 20 should send a HID input report whenever it captures a touch message in the second device 11 locally， wherein the HID input report comprises a data packet containing information about the HID device being used including information about the captured touch message. That is， one touch message by default corresponds with at least one HID input report. The HID input report itself is determined by the HID descriptor which outlines specific information about the usage parameters of the HID device. For example， the HID input report for a HID mouse will provide the maximum and minimum values of the relative X and Y axis.

As given above， the HID input report of the touch screen and HID devices being used to back control the first device 10 is initialized according to the touch message captured by the

Application 20. By default， the

Application 20 doesn’t provide the facility to capture a touch message until

HID is connected and the communication protocol 17 is active， that is， the first device’s screen 12 is cast into the SoC platform.

Although touch message is defined differently in different OS， touch message generally includes a touch point (X， Y) in the touch screen 13b and the touch state， for example， pressed， moving or released.

However， it is impractical for

Application 20 to send a HID input report for every touch message captured. The time it takes to send a HCI Asynchronous Connection-Less (ACL) packet wrapped with the HID input report will invariably be longer than the touch message sample rate in the second device 11 locally， which will lead to an apparent latency when

HID is back-controlling the first device 10. As such， the interval of time between sending HID input reports needs to be considered.

Furthermore， it is difficult to correctly match touch point in the local screen 13c with the touch point in the remote screen 12 of the first device 10， particularly when the user is writing quickly in the local screen 13c. Therefore， the velocity of the pointer movement in the first device 10 should also be taken into consideration.

For example， in an Android phone， there exists a native limitation of pointer sensitivity that is measured based on the speed and acceleration of the pointer. The speed of the pointer in an Android phone has a range of [-7， 7] . A scale factor is applied to the velocity to adapt the resolution of the input device to the resolution of the output device. In this case， the scale factor is applied to adapt the resolution of the local screen 13c to the resolution of the remote screen 12. The first device 10 will by default have a low threshold velocity， which is the scaled speed at which acceleration begins to be applied to the pointer， that is， it is the upper bound of pointer speed at which small motions can be performed without acceleration. In an Android phone， the low threshold of the pointer’s velocity is limited to 500pixel/sec. If the pointer’s velocity exceeds the low threshold， LV， for example， up to 3000 pixel/sec， the Android phone will apply the pointer’s acceleration additionally so that the pointer can move with the extra distance. Consequently， the pointer will become out of the control under such condition. As such， if the pointer’s velocity isn’t taken into consideration， the issue of floating pointer in the first device 10 during quick handwriting can be easily observed.

To address these problems，

Application 20 is arranged to re-sample the captured touch message， and based on the re-sample rate as well as the speed that the pointer in the screen 12 of the first device 10 is moving， periodically send a HID input report to the first device 10. By sending the HID input report to the first device 10 on a periodic basis， for example， one HID input report every 10 to 40 millisecond， the

Application 20 can guarantee that the speed of the pointer’s movement in the remote screen cannot reach the predetermined threshold determined by the first device 10 itself. Consequently， the pointer’s acceleration will not be triggered， which is what cause the pointer to float.

Generally， the native sample rate for capturing the local touch point in the local screen 13c is a static value， for example， 125 Hz. However， since every user will write in the local screen 13c at different speeds， it is desirable for the

Application 20 to be arranged to handle the captured touch messages in a flexible way. Therefore， based on the speed of the user’s input to the local screen 13c， the

Application 20 will re-sample the touch message at a different rate to determine how frequently to send the HID input reports. As such， the re-sample rate is the rate at which the touch message is captured by the

Application 20. For example， RT＝n， whereby RT＝1 indicates that the

Application 20 handles every 1 touch message， RT＝2： indicates that the

Application 20 handles every 2 touch messages and so on.

The bigger the re-sample rate the more difficult it is to make the handwriting recognizablee in the first device 10. If the RT is greater than 1 it is implicitly required that the

Application 20 should send an additional padding report. The padding report is also a HID input report， with the main difference that the fields of X and Y in the padding report are always 0. That is， the padding report doesn’t make the pointer in the first device 10 move at all. Here， the padding report aims at solving the issue of floating pointer in the first device 10 during quick handwriting to some extent. The padding report is added whenever the

Application 20 is going to handle a touch message， while the count for the padding report is limited in the range of [0， RT-1] .

The time， TH， to send HID input report to the

chip 22 is dependent on different

stacks 21， for example， TH may be in the region of 8ms to 16ms. Consequently， the time， TC， for the

chip 22 to send the HID input report to the first device 10 is related to TH. As described above， there is a default sample rate for the capturing local touch message in the local screen， for example， 125Hz or 8ms， as determined by the native limitations of the second device 11. Correspondingly， the sample rate in the local screen determines how many “raw” touch messages will be captured by

Application 20. The

Application 20 will put the local touch messages into a queue， following which it will be re-sampled. The

Application 20 will then send a HID report which has been integrated with the re-sampling into the

stack 21. Then

stack 21 will send a HCI ACL data packet to the

chip 22.

Notably， a

chip 22， such as Cambridge Silicon Radio’s

chip， can always support a “sniff” mode which can guarantee that the HCI ACL data packet is sent to the first device 10 periodically， for example， every 10 -40ms. Based on this feature of

“sniff” mode， the improved remote screen control discussed herein can be implemented. As such， “TC” is determined by the “sniff” mode native to the

chip 22.

As indicated above， the low threshold， LV， of the pointer’s velocity limited in the first device 10 needs to also be considered， a typical low threshold may being， for example， 500pixel/sec.

Taking the above factors into account， the velocity ofthe pointer in the first device 10 may be calculated using the following expression：

wherein (X0， Y0) are the coordinates for the previous local point， P0， the local point being captured in the input/output interface 13 of the second device 11， (X1， Y1) are the coordinates for the current local point， P1， (X0’ ， Y0’ ) are the coordinates for the previous remote point， P0’ ， in the first device 10 which is converted from P0， and (X1’ ， Y1’ ) are the coordinates of the current remote point， P1’ ， in the first device 10 which is converted from P1. As such， D is movement distance of the pointer in the first device 10.

Combining with the analysis above and a practical verification based on smartphones available on the market， the

Application 20 is recommended to send a HID input report under the following conditions：

S＜LV (6)

RT＝3 (7)

Figure 3 provides an example of how the

Application 20 sends a HID input report， while RT is set as 3. As described previously， RT is a dynamic value which is greater or equal to 1. A larger value of RT indicates that more touch messages are ignored， which results in the handwriting controlled by the HID device and shown in the smartphone’s screen is more difficult to recognize. A smaller value of RT indicates that more touch messages are used and sent out in a HID input report. This results in an apparent latency in the first device’s screen. If RT is not calculated properly， the issue of floating pointer in first device 10 can be easily observed. Therefore， RT＝3 is just one example of a suitable value for the re-sample rate.

The

Application 20 first captures a ‘Down’ message 3a which sends a HID input report indicating to the first device 10 that the user has commenced using the

HID. As the user moves the

HID， the

Application 20 captures a plurality of touch messages， M0 to M8. As the re-sample rate is set to 3， the

Application 20 initializes and sends a HID input report for every three touch messages captured. That is， it sends a HID input report for M0 (3b) ， M3 (3c) and M6 (3d) . The

Application 20 also sends a HID input report for the last touch message captured， M8 (3e) . The

Application 20 will finally capture an ‘Up’ message (3f) which sends a HID input report indicating that the user has finished using the

HID， and that there will be no further movement of the pointer.

Given the above description， Figure 4 illustrates the general algorithm implemented by the

Application 20 in order to improve the second device’s 11 ability to back-control the remote screen 12 of the first device， in particular， addressing the common issue of floating pointer. At step A， the

Application 20 captures user input data， that is， touch messages， via the HID device provided in the second device 11 (for example， HID touch screen， HID mouse or HID consumer control as described above) . By default， such user input data is transmitted to the first device 10 in a HID input report every packet of user input data that is captured. Using Bluetooth SPP， the

Application 20 retrieves the remote screen information from the first device 10 to determine the speed， S， at which user input data is received， which is subsequently the speed at which user input data will output to the first device screen 12， that is， the speed at which the pointer in the first devicve 10 moves (step B) . The

Application 20 also establishes the low threshold velocity， LV， of the pointer in the screen of the first device 10 (step B) .

The measured speed， S， is then compared with the low threshold velocity of the smartphone， LV， to see if the user input data is being received and transmitted at speed that is too fast for the first device 10 to handle (step C) . If the speed， S， is less than the low threshold velocity， LV， of the first device 10， the

Application 20 will transmit the user input data to the first device 10 at its initial transmission rate (step D) ， that is， it will continue to transmit the user input data every time user input data is captured. If the speed， S， is more than the low threshold velocity， LV， of the first device 10， the

Application 20 will re-sample the user input data (step E) in dependence on the low threshold velocity， LV， and from the re-sampled rate determine a second transmission rate (step F) . For example， the low threshold velocity for an Android phone is 500 pixels per second， and therefore the recommended re-sample rate is 3， whereby user input data is transmitted to the first device 10 for every 3 lots of user input data received， as illustrated by Figure 3. The

Application 20 will then transmit the user input data to the smartphone (step G) at the established second transmission rate. The

Application 20 will continue to perform steps A-G until no further user input data is being captured.

An overview of the system described above is illustrated by Figure 5. Once the

HID is connected and a link has been established between the first device 10 and the Smartphone Link Application 23 (steps 1 and 2) ， the

SPP is connected between the

Application 20 and the first device 10 (step 3) . If the SPP connection is terminated， the

Application 20 is responsible for recovering the SPP connection if the

HID is still connected.

Steps 4 to 6 are used to control the first device 10 through a

HID or consumer control， wherein HID input reports are sent from the

Application 20 to the first device 10 based on the rate that the touch message is captured as well as the speed of the pointer in the first device 10.

Once the user has finished and the

HID has been disconnected (step 7) ， the

Application 20 disconnects the

SPP automatically (step 8) .

Various modifications， whether by way of additions， deletion or substitution may be made to the arrangements described herein， any and all of which are intended to be encompassed by the invention as defined by the appended claims.

Claims

A method of communicating user inputs between a first device and a second device， the method being performed by the first device and comprising：

detecting one or more user inputs at the first device；

comparing a speed of the one or more user inputs with a rate of communications between the first device and the second device；

sampling user input data corresponding to the one or more user inputs based at least in part on the comparison； and

transmitting the user input data to the second device using a first communication protocol.
The method of claim 1， wherein the first communication protocol includes a
Human-Interface Device (HID) profile.
The method of claim 1， wherein the first communication protocol includes a
Serial Port Profile (SPP) .
The method of claim 1， wherein the sampling comprises：

sampling the user input data at a first rate； and

re-sampling the user input data at a second rate if the speed of the one or more user inputs is greater than the rate of communications.
The method of claim 4， wherein a rate of the re-sampling is based at least in part on the rate of communications between the first device and the second device.
The method of claim 1， further comprising：

retrieving information from the second device indicating the rate of communications.
The method of claim 1， wherein the transmitting comprises：

determining a transmission rate for the user input data based at least in part on the speed of the one or more user inputs.
The method of claim 1， wherein the user input data is periodically transmitted to the second device.
The method of claim 1， wherein the rate of communications is at most 500 pixels/sec.
A computing device， comprising：

one or more processors； and

a memory storing instructions that， when executed by the one or more processors， cause the computing device to：

detect one or more user inputs；

compare a speed of the one or more user inputs with a rate of communications between the computing device and a target device；

sample user input data corresponding to the one or more user inputs based at least in part on the comparison； and

transmit the user input data to the target device using a first communication protocol.
The computing device of claim 10， wherein the computing device is an in-vehicle communication device.
The computing device of claim 10， further comprising a touch screen， a Human Interface Device (HID) mouse and/or a HID consumer control.
The computing device of claim 10， wherein the first communication protocol includes a
Human-Interface Device (HID) profile.
The computing device of claim 10， wherein the first communication protocol includes a
Serial Port Profile (SPP) .
The computing device of claim 10， wherein execution of the instructions to sample the user input data causes the computing device to：

sample the user input data at a first rate； and

re-sample the user input data at a second rate if the speed of the one or more user inputs is greater than the rate of communications.
The computing device of claim 15， wherein a rate of the re-sampling is based at least in part on the rate of communications between the computing device and the target device.
The computing device of claim 10， wherein execution of the instructions further causes the computing device to：

retrieve information from the target device indicating the rate of communications.
The computing device of claim 10， wherein execution of the instructions to transmit the user input data causes the computing device to：

determine a transmission rate for the user input data based at least in part on the speed of the one or more user inputs.
The computing device of claim 10， wherein the user input data is periodically transmitted to the target device.
A non-transitory computer-readable medium storing instructions that， when executed by one or more processors of a computing device， cause the computing device to：

detect one or more user inputs；

compare a speed of the one or more user inputs with a rate of communications between the computing device and a target device；

sample user input data corresponding to the one or more user inputs based at least in part on the comparison； and

transmit the user input data to the target device using a first communication protocol.