CN103403649B - User input back channel for wireless displays - Google Patents

Abstract

As part of a communication session, a wireless source device can transmit audio and video data to a wireless sink device, and the wireless sink device can transmit user input data received at the wireless sink device back to the wireless source device. In this manner, a user of the wireless sink device can control the wireless source device and control the content that is being transmitted from the wireless source device to the wireless sink device. The input data received at the wireless sink device can have associated coordinate information that is scaled or normalized by either the wireless sink device or the wireless source device.

Description

User input Return Channel for radio display

This application claims the rights and interests of following U.S. Provisional Application：

The U.S. Provisional Application No.61/435 submitting on January 21st, 2011,194；

The U.S. Provisional Application No.61/447 submitting for 2 months 28 days in 2011,592；

The U.S. Provisional Application No.61/448 submitting on March 2nd, 2011,312；

The U.S. Provisional Application No.61/450 submitting on March 7th, 2011,101；

The U.S. Provisional Application No.61/467 submitting on March 25th, 2011,535；

The U.S. Provisional Application No.61/467 submitting on March 25th, 2011,543；

The U.S. Provisional Application No.61/514 submitting for 3rd in August in 2011,863；

The U.S. Provisional Application No.61/544 submitting on October 7th, 2011,440；

With way of reference, the full content of above-mentioned provisional application is expressly incorporated herein.

Technical field

Present disclosure relates to send the technology of data between radio source device and wireless host device.

Background technology

Wireless Display（WD）Or Wi-Fi shows（WFD）System includes radio source device and one or more wireless host device. Source device and each host device can be to have mobile device or the wireline equipment of wireless communication ability.For example, source device and One or more of host device can include mobile phone, the portable computer with wireless communication card, individual digital help Reason（PDA）, portable electronic device or other there is this equipment of wireless communication ability, including so-called " intelligent " Phone and " intelligent " flat board or panel computer, electronic reader or any kind of radio display, video game device or its The Wireless Telecom Equipment of its type.One or more of source device and host device can also include thering is the wired of communication capacity Equipment, TV, desk computer, monitor, projector etc..

Source device shares one or more of the host device of session host device transmission media data to participating in specific medium （Such as audio frequency and video（AV）Data）.Can show in each of display of the local display of source device and host device Show and at device, media data is played back.More properly, each of host device being participated in host device is in its screen and sound Received media data is presented on frequency equipment.

Content of the invention

Present disclosure briefly describes the system that wireless host device can be communicated wherein with wireless host device. As a part for communication session, radio source device can send Voice ＆ Video data to wireless host device, and wireless place The user input receiving at this wireless host device can be sent back radio source device by equipment.By this way, wireless place sets Standby user can control radio source device and can control the content sending to wireless host device from radio source device.

In one example, a kind of method sending user data to radio source device from wireless host device includes：Institute State acquisition user input data at wireless host device, wherein, described user input data has associated coordinate data；To institute State associated coordinate data to be normalized, to generate normalized coordinate data；Generate and include described normalized coordinate The packet of data；And send described packet to radio source device.

In another example, a kind of wireless host device for sending user data to radio source device includes：Storage The memorizer of instruction；One or more processors, it is configured to execute described instruction, wherein, after executing described instruction, described One or more processors make：User input data, wherein, described user input data is obtained at described wireless host device There is associated coordinate data；Described associated coordinate data is normalized, to generate normalized coordinate data； Generate the packet including described normalized coordinate data；And transmission unit, it is used for sending out to described radio source device Send described packet.

In another example, a kind of computer-readable recording medium of store instruction, in described instruction by one or many So that one or more of computing device sends number of users from wireless host device to radio source device after individual computing device According to method.Methods described includes：User input data, wherein, described user input number is obtained at described wireless host device According to having associated coordinate data；Described associated coordinate data is normalized, to generate normalized number of coordinates According to；Generate the packet including described normalized coordinate data；And send described packet to radio source device.

In another example, a kind of wireless host device for sending user data to radio source device includes：For Obtain the module of user input data at described wireless host device, wherein, described user input data has associated seat Mark data；For being normalized to described associated coordinate data, to generate the module of normalized coordinate data；For Generate the module of the packet including described normalized coordinate data；And for sending described data to radio source device The module of packet.

In another example, a kind of include from the method for wireless host device receive user data at radio source device： Receiving data packet at described radio source device, wherein, the user that described packet includes being associated with coordinate data is defeated Enter data；Described associated coordinate data is normalized, to generate normalized coordinate data；And returned based on described The coordinate data of one change is processed to described packet.

In another example, a kind of include for the radio source device from wireless host device receive user data：Transmission Unit, it is used for receiving data packet at described radio source device, and wherein, described packet includes related to coordinate data The user input data of connection；Memorizer, its store instruction；One or more processors, it is configured to execute described instruction, its In, after executing described instruction, one or more of processors make：Normalizing is carried out to described associated coordinate data Change, to generate normalized coordinate data；And based on described normalized coordinate data, described packet is processed.

In another example, a kind of computer-readable recording medium of store instruction, in described instruction by one or many After individual computing device so that one or more of computing device at radio source device from wireless host device receive user The method of data.Methods described includes：Receiving data packet at described radio source device, wherein, described packet includes The user input data being associated with coordinate data；Described associated coordinate data is normalized, to generate normalization Coordinate data；And based on described normalized coordinate data, described packet is processed.

In another example, a kind of include for the radio source device from wireless host device receive user data：For The module of receiving data packet at described radio source device, wherein, described packet includes being associated with coordinate data User input data；For being normalized to described associated coordinate data, to generate the mould of normalized coordinate data Block；And the module for being processed to described packet based on described normalized coordinate data.

Brief description

Figure 1A is the block diagram of the example of source/place system illustrating to realize the technology of present disclosure.

Figure 1B is the block diagram of the example of source/place system illustrating to have two host devices.

Fig. 2 shows the example of the source device of the technology that can realize present disclosure.

Fig. 3 shows the example of the host device of the technology that can realize present disclosure.

Fig. 4 shows the transmitter system of the technology that can realize present disclosure and the block diagram of receiver system.

Fig. 5 A and 5B shows that the technology according to present disclosure is used for the example message transmission sequence that executive capability is consulted.

Fig. 6 shows that the sample data that can be used for delivering, to source device, the user input data obtaining at host device divides Group.

Fig. 7 A and 7B is the technology illustrating can be used for the present disclosure of the capability negotiation between source device and host device Flow chart.

Fig. 8 A and 8B is in the disclosure illustrating to can be used for send and receive the packet with user input data The flow chart of the technology held.

Fig. 9 A and 9B is in the disclosure illustrating to can be used for send and receive the packet with user input data The flow chart of the technology held.

Figure 10 A and 10B is to illustrate to can be used for sending and receiving the data with timestamp information and user input data The flow chart of the technology of present disclosure of packet.

Figure 11 A and 11B is to illustrate to can be used for sending and receiving the data with timestamp information and user input data The flow chart of the technology of present disclosure of packet.

Figure 12 A and 12B is the present disclosure illustrating to can be used for sending and receiving the packet including voice command Technology flow chart.

Figure 13 A and 13B is to illustrate to can be used for sending and receiving the packet with multiple point touching user input order The technology of present disclosure flow chart.

Figure 14 A and 14B is to illustrate to can be used for sending and receiving have the user input data forwarding from third party device The technology of the present disclosure of packet flow chart.

Figure 15 A and 15B is the flow chart illustrating to can be used for the technology of the present disclosure sending and receiving packet.

Specific embodiment

Present disclosure briefly describes the system that wireless host device can be communicated with wireless host device.As logical A part for letter session, radio source device can send Voice ＆ Video data to wireless host device, and wireless host device can So that the user input receiving at this wireless host device to be sent back this radio source device.By this way, wireless host device User can control radio source device, and can control the content sending from radio source device to wireless host device.

Figure 1A is the exemplary source/place system illustrating to realize one or more of technology of present disclosure technology 100 block diagram.As shown in Figure 1A, system 100 includes the source device being communicated via communication channel 150 with host device 160 120.Source device 120 can include storing audio/video（A/V）The memorizer of data 121, display 122, speaker 123, sound Frequently/video encoder 124（Also referred to as encoder 124）, audio/video control module 125 and emittor/receiver（TX/ RX）Unit 126.Host device 160 can include display 162, speaker 163, audio/video decoder 164（Also referred to as solve Code device 164）, emittor/receiver unit 166, user input（UI）Equipment 167 and user input processing module（UIPM）168. Shown assembly merely comprises an example arrangement of source/place system 100.Other configurations can be included than those shown groups The less assembly of the part or other assembly outside those shown assemblies can be included.

In the example of Figure 1A, source device 120 can show the video portion of audio/video data 121 on display 122 Point, and the audio-frequency unit of audio/video data 121 can be exported on speaker 123.Audio/video data 121 can this Be stored on source device 120, from exterior storage medium（Such as file server, hard disk drive, external memory storage, blue light CD, DVD or other physical storage medium）Access or can set via the network connection source of transmitting as a stream of such as the Internet Standby 120.In some cases, via the photographing unit of source device 120 and mike, reality can be carried out to audio/video data 121 When catch.Audio/video data 121 can include the content of multimedia of such as film, TV programme or music etc, but also may be used To include the real time content being generated by source device 120.For example, this real time content can be by operating on source device 120 Application produces, or the video data capturing（For example as a part for video-phone session）.As will be more detailed Thin description, in some cases, this real time content can include being available for the frame of video of the user input option of user's selection. In some cases, audio/video data 121 can include the frame of video of the combination of dissimilar content, such as has covering The frame of video of the movie or television program of the user input option in frame of video.

In addition to locally assuming audio/video data 121 via display 122 and speaker 123, source device 120 Audio/video encoder 124 audio/video data 121 can be encoded, and emittor/receiver unit 126 can To send encoded data to host device 160 in communication channel 150.The emittor/receiver unit 166 of host device 160 Receive this encoded data, and audio/video decoder 164 is decoded to this encoded data, and via display Device 162 and speaker 163 export decoded data.By this way, the audio frequency being presented by display 122 and speaker 123 Can be presented by display 162 and speaker 163 with video data simultaneously.Voice data and video data can be arranged in frame In, and when present, audio frame can be synchronous with video frame time.

Audio/video encoder 124 and audio/video decoder 164 can realize any number of Voice ＆ Video pressure Contracting standard, such as ITU-T H.264 standard（Or referred to as MPEG-4, part 10）, advanced video coding（AVC）Or it is emerging High efficiency Video coding（HEVC）Standard（It is sometimes referred to as H.265 standard）.Can also using many other types of privately owned or Standardized compress technique.Generally speaking, audio/video encoder 164 is configured to execute the mutual of audio/video encoder 124 Easily encoding operation.Although not illustrating in figure ia, in some respects, A/V encoder 124 and A/V decoder 164 are all permissible Integrated with audio coder and decoder it is possible to include suitable MUX-DEMUX（Multiplexing-demultiplexing）Unit or other hardware And software, to process the coding to both the Voice ＆ Videos in common data stream or separate data stream.

As following will be described in more detail, in addition to realizing video compression standard as above, A/V encode Device 124 can also carry out other encoding functions.For example, before A/V data 121 is sent to host device 160, A/V encodes Device 124 can add various types of metadata to A/V data 121.In some cases, can be in an encoded form by A/V Data 121 is stored in source device 120 or receives A/V data 121, from without by A/V encoder at source device 120 124 compress further.

Although Figure 1A shows the communication channel individually carrying audio payload data and video payload data 150, but it is to be understood that, in some cases, video payload data and audio payload data can be public A part for data flow.If be suitable for, MUX-DEMUX unit can follow ITU H.223 Multiplexer Protocol or such as user Datagram protocol（UDP）Etc other agreements.Audio/video encoder 124 and audio/video decoder 164 all can be real It is now one or more microprocessors, digital signal processor（DSP）, special IC（ASIC）, field-programmable gate array Row（FPGA）, discreet logic device, software, hardware, firmware or its combination in any.Can be by audio/video encoder 124 and sound Frequently each of/Video Decoder 164 includes in one or more encoders or decoder, can be by any of which one It is integrated into the encoder/decoder of combination（CODEC）A part.Therefore, each of source device 120 and host device 160 May be configured to execute the special purpose machinery of one or more of the technology of present disclosure technology.

Display 122 and display 162 can include any one of various picture output devices, such as cathode ray Pipe（CRT）, liquid crystal display（LCD）, plasma display, light emitting diode（LED）Display, Organic Light Emitting Diode （OLED）Display or other types of display device.In these or other examples, display 122 and 162 may each be to be sent out Penetrate formula display or transmissive display.Display 122 and 162 can also be touch display so that they are defeated simultaneously Entering equipment is also display device.This kind of touch display screen can be condenser type, resistance-type or allow user to carry to respective equipment Other types of touch panel for user input.

Speaker 123 can include any one of various audio output apparatus, such as headband receiver, single speaker System, multi-loudspeaker system or ambiophonic system.In addition although display 122 and speaker 123 are shown as source device 120 A part a part of and that display 162 and speaker 163 are shown as host device 160, but source device 120 and host device 160 Can essentially be device systems.As an example, display 162 can be television set, and speaker 163 can be surround sound System, and decoder 164 can be a part for the outer container being wired or wirelessly connected to display 162 and speaker 163. In other cases, host device 160 can be the individual equipment of such as panel computer or smart phone etc.In other situations Under, source device 120 is similar equipment with host device 160, and for example, the two is all smart mobile phone, panel computer etc..In this feelings Condition, an equipment can be operated as source, and another equipment can be operated as place.In follow-up communication In session, these roles can even invert.In other cases, source device can include such as smart phone, meter on knee The mobile device of calculation machine or panel computer etc, and host device can include more static equipment（For example, there is AC power supplies Line）, in this case, source device can be via host device transmission Voice ＆ Video data to larger crowd to be in Existing.

Emittor/receiver unit 126 and emittor/receiver unit 166 may each comprise various frequency mixers, filtering Device, amplifier and the other assemblies being designed for signal modulation, and one or more antenna and be designed for sending and receiving Other assemblies of data.Communication channel 150 generally represents for sending appointing of video data from source device 120 to host device 160 What suitable communication media or the set of different communication medium.Communication channel 150 is typically the communication channel of relatively short distance, class It is similar to Wi-Fi, bluetooth etc..However, communication channel 150 is not necessarily limited in this respect, and can include any wirelessly or non-wirelessly Communication media（Such as radio frequency（RF）Frequency spectrum or one or more physical transmission line）Or wireless and wire medium combination in any. In other examples, communication channel 150 even can form packet-based network（Such as wired or wireless LAN, wide The global network of domain net or such as the Internet）A part.Additionally, communication channel 150 can be by source device 120 and host device 160 are used for creating peer link.Source device 120 and host device 160 can be using the marks being such as derived from IEEE 802.11 family of standards Accurate communication protocol is communicated in communication channel 150.For example, source device 120 and host device 160 can be straight according to Wi-Fi The standard of connecing is communicated so that source device 120 and host device 160 are in the matchmaker not using such as WAP or so-called focus Directly communicate each other in the case of Jie.Source device 120 and host device 160 can also be set up tunnel type direct link and set up （TLDS）To avoid or to reduce network congestion.Sometimes it is referred to Wi-Fi to describe the technology of present disclosure, but it should envisioning , each side of these technology can also be compatible with other communication protocols.Infinite mode by way of example, source device Radio communication between 120 and host device can use orthogonal frequency division multiplexi（OFDM）Technology.Can also be using various each Other wireless communication technologys of sample, including but not limited to time division multiple acess（TDMA）, frequency division multiple access（FDMA）, CDMA（CDMA） Or the combination in any of OFDM, FDMA, TDMA and/or CDMA.WiFi is directly and TDLS is intended to set up the communication meeting of relatively short distance Words.In this context, relatively short distance can refer to e.g., less than 70 meters, although noisy or stop environment in, equipment it Between distance possibly even shorter, all such as less than 35 meters.

Except to from the decoding data that source device 120 receives with addition to presenting, host device 160 can also be defeated from user Enter equipment 167 receiving user's input.For example, user input device 167 can be keyboard, mouse, trace ball or track pad, touch Screen, voice command recognition module or any other this user input device.UIPM will be received by user input device 167 User input command format is melted into the data packet structure that source device 120 can be explained.By emittor/receiver 166 in communication On channel 150, these packets are sent to source device 120.Emittor/receiver unit 126 receives this packet, and And A/V control module 125 is parsed to this packet and is ordered by the user input that user input device 167 receives with explaining Order.Based on the order receiving in packet, A/V control module 125 can change the content being encoded and being sent.With this The mode of kind, the user of host device 160 can be remotely effective to the audio payload data sending and video by source device 120 Load data is controlled, and indirectly interacts with source device 120.The user of host device 160 can be to source device 120 The example of the type of order sending includes for being refunded, the order of F.F., time-out and broadcasting Voice ＆ Video data, Yi Jiyong Order in scaling, rotation, rolling etc..For example, user can also be selected from options menu, and this selection is sent Return source device 120.

Additionally, the user of host device 160 can start and control the application on source device 120.For example, host device 160 User can start the photo editing application being stored on source device 120, and using this application to being locally stored in source device Photo on 120 enters edlin.Host device 160 can present to user and seems and be sensuously right in host device 160 The Consumer's Experience that picture is locally edited, and actually on source device 120, edlin is entered to picture.Joined using such Put, the ability of an equipment can be used for some equipment by equipment user.For example, source device 120 can be to have a large amount of storages Device and the smart phone of high-end disposal ability.The user of source device 120 can be in the commonly used all settings of smart phone With in situation using this smart phone.However, when watching film, user may want to there is the equipment compared with large display screen Upper viewing film, in this case, host device 160 can be panel computer or even larger display device or TV Machine.When wanting transmission or reply email, user may want to using the equipment with keyboard, in this case, place Equipment 160 can be laptop computer.In above-mentioned two situations, although user is interacted with host device, greatly Most process can be still by source device 120（It is smart phone in this example）To execute.In this specific operation context In, because great majority operation is executed by source device 120, if therefore require host device 160 to execute just entered by source device 120 The process of row, then host device 160 can be to have the equipment of the lower cost of fewer resource by contrast.In some instances, Both source device and host device can accept user input（Such as touch screen order）, and the technology of present disclosure is permissible By holding consultation and/or identifying to promote two-way interactive to the ability of equipment in any given session.

In some configurations, A/V control module 125 can be the operation system of the operating system execution just by source device 125 System process.However, in other configurations, A/V control module 125 can be the software mistake operating in the application on source device 120 Journey.In this configuration, user input order can be explained by software process, so that the user of host device 160 is direct Interact with operating in applying rather than operating in the operating system on source device 120 on source device 120.By directly with Application rather than operating system interact, and the user of host device 160 can not be the operating system of source device 120 with Internet access Local command library.Additionally, directly interacting so that ordering the equipment being running in different platform with application More easily it is transmitted and process.

Source device 120 can respond to the user input of application at wireless host device 160.Should in this interactive mode With, in setting, the user input of application at wireless host device 160 being sent back Wireless Display in communication channel 150 Source.In one example, it is possible to achieve backward channel framework（Also known as user interface Return Channel（UIBC））So that place sets Standby 160 user inputs that can be sent in application at host device 160 to source device 120.Backward channel framework can include for The upper layer message of transmission user input and for holding consultation to the user interface capabilities at host device 160 and source device 120 Lower layer frame.UIBC may be located at the Internet Protocol between host device 160 and source device 120（IP）Transport layer.By this way, UIBC can be in Open System Interconnection（OSI）On transport layer in traffic model.In one example, OSI communication includes seven Layer（1- physics, 2- data link, 3- network, 4- transmission, 5- session, 6- represent and 7- application）.In this example, in transport layer On refer to layer 5,6 and 7.In order to lift reliable transmission and the order of the packet comprising user input data be passed Send, UIBC is configurable in other packet-based communication protocols（Such as transmission control protocol/Internet Protocol（TCP/IP） Or UDP（UDP））Top run.UDP and TCP can parallel work-flow in osi layer framework.TCP/IP is permissible Host device 160 and source device 120 is enable to realize retransmission technique in case of packet loss.

In some cases, it can be possible to lose existing between the user input interface at source device 120 and host device 160 Join.In order to solve the potential problems that led to by this mismatch and lift good Consumer's Experience, Ke Yi in these cases Before setting up communication session or in each moment running through communication session, used between source device 120 and host device 160 Family input interface capability negotiation.As a part for this negotiations process, source device 120 and host device 160 can be with regard to the screens consulted Curtain resolution is reached an agreement.When host device 160 sends the coordinate data being associated with user input, host device 160 can be right The coordinate data obtaining from display 162 zooms in and out, to mate consulted screen resolution.In one example, if Host device 160 has the resolution of 1280x720, and source device 120 has the resolution of 1600x900, then equipment can be such as Resolution that 1280x720 as it consult is used.Although can also be using the resolution of source device 120 or some other points Resolution, but can be based on the resolution of host device 160 to select the resolution consulted.In the host device using 1280x720 In example, host device 160 can be sat with 1600/1280 x being obtained by factor pair before sending coordinate to source device 120 Mark zooms in and out, and similarly, and host device 160 can be before sending coordinate to source device 120, with 900/720 to being obtained The y-coordinate obtaining zooms in and out.In other configurations, the coordinate of acquisition can be zoomed to consulted resolution by source device 120. Zooming in and out can be whether to use the resolution display higher than source device 120 based on host device 160, to increase or decrease Coordinate range, or vice versa as the same.

Additionally, in some instances, the resolution at host device 160 may change during communication session, thus Potentially result in the mismatch between display 122 and display 162.In order to improve Consumer's Experience and guarantee suitable function, Source/place system 100 can be by realizing realizing for reducing or preventing user mutual mismatch for the normalized technology of screen Technology.The display 162 of the display 122 of source device 120 and host device 160 can have different resolution and/or not Same aspect ratio.Additionally, in some settings, the user of host device 160 can have for from source device 120 reception Video data is adjusting the ability of display window size, so that covering the whole of the display 162 less than host device 160 The video data receiving from source device 120 is assumed in window.In the setting of another kind of example, the user of host device 160 can have There is the selection watching content with transverse mode or vertical pattern, each pattern has coordinate and the high width of different screens of uniqueness Than.In these cases, the coordinate being associated with the user input receiving at host device 160（Such as click or touch The coordinate of event point）Possibly cannot be processed in the case of coordinate not being modified by source device 120.Cause This, the technology of present disclosure can include being mapped to and source device the coordinate of the user input receiving at host device 160 120 associated coordinates.Herein, this mapping is also known as normalization, and this mapping will be carried out more detailed below Explain, this mapping can be based on place or based on source.

The user input being received by host device 160 can be received by UI module 167（For example, in driver layer）, and quilt Pass to the operating system of host device 160.Operating system in host device 160 can receive, with display surface, user occur The coordinate that input is associated（x_SINK, y_SINK）.In this illustration,（x_SINK, y_SINK）Can be that click or touch occur The coordinate of the display 162 at event.The display window being presented on display 162 can have the big of this display window of description Little x- coordinate length（L_DW）With y- coordinate width（W_DW）.This display window can also have the position describing this display window Top left co-ordinate（a_DW, b_DW）.Based on L_DW、W_DWAnd top-left coordinates（a_DW, b_DW）It may be determined that display 162 is shown window The part covering.For example, the upper right corner of display window may be located at coordinate（a_DW+L_DW, b_DW）, the lower left corner of display window is permissible Positioned at coordinate（a_DW+L_DW, b_DW+W_DW）, and the lower right corner of display window may be located at coordinate（a_DW+L_DW, b_DW+W_DW）.If input It is to receive at the coordinate in display window, then this input can be processed by host device 160 as UIBC input.Change Sentence talk about, if meeting following condition, can by with coordinate（x_SINK, y_SINK）Associated input as UIBC input into Row is processed：

a_DW≤x_SINK≤a_DW+L_DW（1）

b_DW≤y_SINK≤b_DW+W_DW（2）

Determine user input be UIBC input after, before being sent to source device 120, UIPM168 can to The coordinate that this input is associated is normalized.The input being determined to be outside display window can be local by host device 160 Processed as non-UIBC input.

As mentioned above, the normalization of input coordinate can be based on source or based on place.When realization is based on During the normalization in source, the display resolution that display 122 can be supported by source device 120（L_SRC, W_SRC）With video data one Rise or be sent to host device 160 independent of video data.For example, it is possible to using the display resolution supported as capability negotiation meeting A part for words is transmitted or can be transmitted in another moment during communication session.Host device 160 can be true Determine the display resolution of display 162（L_SINK, W_SINK）, the display window of window of content that receives from source device 120 of display divides Resolution（L_DW, W_DW）, and this display window top left co-ordinate（a_DW, b_DW）.As described above, it is relative with user input when determining The coordinate answered（x_SINK, y_SINK）When in display window, the operating system of host device 160 can be using transfer function by coordinate （x_SINK, y_SINK）It is mapped to source coordinate（x_SRC, y_SRC）.For inciting somebody to action（x_SINK, y_SINK）It is converted into（x_SRC, y_SRC）Example conversion letter Number can be as follows：

x_SRC=（x_SINK-a_DW）*（L_SRC/L_DW）（3）

y_SRC=（y_SINK-b_DW）*（W_SRC/W_DW）（4）

Therefore, when the coordinate that the user input sending with receive is corresponding, host device 160 can be sent in（x_SINK, y_SINK）The coordinate of the user input that place receives（x_SRC, y_SRC）.As described in more detail in will be detailed below, for example, it is possible to will sit Mark（x_SRC, y_SRC）Divide as the data for being sent in, to source device 120, the user input receiving at host device 160 on UIBC Group a part of being transmitted.Run through other portions input coordinate being described as including the present disclosure in packet Point, as realized described in the normalized example based on place in source/place system 100 above, those coordinates can be turned Change source coordinate into.

When realizing the normalization based on place when source/place system 100, it is UIBC input rather than local input for determining User input（That is, in display window rather than outside display window）, can be at source device 120 rather than in host device Execution calculating above at 160.In order to promote these calculating, host device 160 can send L to source device 120_DW、W_DWValue and The positional information of display window（For example, a_DW、b_DW）, and（x_SINK, y_SINK）Coordinate.The value being sent using these, source sets Standby 120 can be according to formula above 3 and 4 determining（x_SRC, y_SRC）Value.

In the normalized other realizations based on place, host device 160 can send description user input event and occur to exist The coordinate of the position in display window rather than the generation of the user input event user input of position on display 162（x_DW, y_DW）.In such an implementation, can be by coordinate（x_DW, y_DW）Together with（L_DW, W_DW）Value be sent to source device 120.Based on these The value receiving, source device 120 can determine according to following transfer function（x_SRC, y_SRC）：

x_SRC=x_DW*（L_SRC/L_DW）（5）

y_SRC=y_DW*（W_SRC/W_DW）（6）

Host device 160 can determine x based on lower array function_DWAnd y_DW：

x_DW=x_SINK-a_DW（7）

y_DW=y_SINK-b_DW（8）

When present disclosure description sends, for example in packet, the coordinate being associated with user input, these coordinates Transmission can include the normalization based on place or based on source as above, and/or can include for execution based on place or Based on extraneous information any necessary to the normalization in source.

UIBC can be designed as transmitting various types of user input datas, including cross-platform user input data.Example As source device 120 can run iOSOperating system, and host device 160 runs such as AndroidOr WindowsEtc Another operating system.Whatsoever platform, UIPM 168 can be with the intelligible form of A/V control module 125 to reception User input is packaged.UIBC can support the user input format of number of different types, to allow many dissimilar Source device and host device utilize agreement, regardless of whether whether source device and host device operate on different platforms.Can define Universal input form, and can simultaneously support the pattern of the input specific to platform, thus with by UIBC in source device 120 He The mode transmitting user input between host device 160 provides motility.

In the example of Figure 1A, source device 120 can include smart phone, panel computer, laptop computer, desktop type Computer, the TV supporting Wi-Fi or any other equipment that Voice ＆ Video data can be sent.Host device 160 can be same Sample ground includes smart phone, panel computer, laptop computer, desktop computer, supports the TV of Wi-Fi or can connect Radio reception frequency and any other equipment of video data and receiving user's input data.In some cases, host device 160 is permissible System including equipment so that display 162, speaker 163, UI equipment 167 and A/V encoder 164 all be partially separated but It is the equipment of interoperable.Source device 120 can be equally the system of equipment, rather than individual equipment.

In this disclosure, term source device is generally used for the equipment referring to send audio/video data, and term place Equipment is generally used for the equipment referring to receive audio/video data from source device.In many cases, source device 120 and host device 160 can be similar or identical equipment, and equipment is operated as source and another equipment operates as Su Jinhang.This Outward, in different communication sessions, these roles can overturn.Therefore, host device in a communication session can be later Become source device in communication session, or vice versa as the same.

Figure 1B is the block diagram of the exemplary source/place system 101 illustrating to realize the technology of present disclosure.Source/place system System 101 includes source device 120 and host device 160, and each of source device 120 and host device 160 can be with above for figures Mode described by 1A is run and is operated.Source/place system 101 also includes host device 180.Host device 180 can with retouch above Host device 160 similar mode stated receives Voice ＆ Video data from source device 120 on the UIBC setting up, and Xiang Yuanshe Standby 120 transmission user commands.In some configurations, host device 160 and host device 180 can be operated independently of one another, and And the Voice ＆ Video data of output can export at host device 160 and host device 180 at source device 120 simultaneously.Replacing In generation configuration, host device 160 can be home equipment, and host device 180 can be auxiliary host device.In this example arrangement, Host device 160 and host device 180 can be coupled, and host device 160 can show video data and host device 180 output phase Corresponding voice data.Additionally, in some configurations, host device 160 can only export the video data of transmission, and host device 180 voice datas that can only export transmission.

Fig. 2 is the block diagram of an example illustrating source device 220.Source device 220 can be and the source device 120 in Figure 1A Similar equipment, and can be to be operated with source device 120 identical mode.Source device 220 includes local display 222nd, local speaker 223, processor 231, memorizer 232, transmission unit 233 and radio modem 234.In Fig. 2 Shown, source device 220 can include A/V data being encoded and/or being decoded to be transmitted, store and to show Or multiple processor（That is, processor 231）.For example, it is possible to by A/V data storage at memorizer 232.Memorizer 232 is permissible Store complete A/V file, or can include only storing（For example, transmit as a stream from another equipment or source）A/V file The less buffer of a part.Transmission unit 233 can be processed for carrying out network biography to encoded A/V data Defeated.For example, encoded A/V data can be processed by processor 231, and be packaged into network by transmission unit 233 and connect Enter layer（NAL）Unit is to carry out internetwork communication.Can by radio modem 234 via network connection by NAL unit It is sent to wireless host device.Radio modem 234 can be for example to be arranged for carrying out one of IEEE 802.11 family of standards Wi-Fi modem.

Source device 220 can also carry out processing locality and display to A/V data.Specifically, video-stream processor 235 is permissible The video data to show in local display 222 is processed, audio process 236 can be to voice data at Reason is to export on speaker 223.

As explained above with described by the source device 120 of Figure 1A, source device 220 can also be defeated from host device receive user Enter.By this way, the data of the encapsulation that the radio modem 234 of source device 220 receives such as NAL unit etc is divided Group, and the data cell that this is encapsulated is sent to transmission unit 233 to be decapsulated.For example, transmission unit 233 is permissible Extract packet from NAL unit, and processor 231 can be parsed to this packet to extract user input life Order.Based on this user input order, processor 231 can be to the encoded A/V data being sent to host device by source device 220 It is adjusted.In this way it is possible to completely or partially realize controlling mould above with reference to the A/V of Figure 1A by processor 231 Function described by block 125.

The processor 231 of Fig. 2 generally represents any one of various processors, including but not limited to one or more numbers Word signal processor（DSP）, general purpose microprocessor, special IC（ASIC）, field programmable logic array（FPGA）, its Its equivalent integrated or discrete logic circuitry or its certain combination.The memorizer 232 of Fig. 2 can include various volatibility or non- Any one of volatile memory, including but not limited to such as Synchronous Dynamic Random Access Memory（SDRAM）Etc Random access memory（RAM）, read only memory（ROM）, nonvolatile RAM（NVRAM）, electric erasable can Program read-only memory（EEPROM）, flash memory etc..Memorizer 232 can be included for storing audio/video data and other The computer-readable recording medium of categorical data.Memorizer 232 can additionally store by processor 231 as the execution disclosure A part for various technology described in content and the instruction that executes and program code.

Fig. 3 shows an example of host device 360.Host device 360 can be similar with the host device 160 in Figure 1A Equipment, and can be to be operated with host device 160 identical mode.Host device 360 includes one or more processors （That is, processor 331）, memorizer 332, transmission unit 333, radio modem 334, video-stream processor 335, locally displayed Device 362, audio process 336, speaker 363 and user input interface 376.Host device 360 is in radio modem 334 Place receives the data cell of the encapsulation sending from source device.Radio modem 334 can be for example to be arranged for carrying out IEEE The Wi-Fi modem of one or more of 802.11 family of standards standard.Transmission unit 333 can be to the data sheet of encapsulation Unit is decapsulated.For example, transmission unit 333 can extract encoded video data from the data cell of encapsulation, and By encoded A/V data is activation to processor 331 to be decoded and to present, to export.Video-stream processor 335 can be right The decoded video data to show in local display 362 is processed, and audio process 336 can be to through solution The voice data of code is processed for output on speaker 363.

In addition to assuming Voice ＆ Video data, wireless host device 360 can also be received by user input interface 376 User input data.User input interface 376 can represent any one in multiple user input devices, including but do not limit In：Touch display interface, keyboard, mouse, speech command module, gesture-capture equipment（For example, there is the input based on photographing unit Capturing ability）Or any other equipment in multiple user input devices.Can be connect to by user input by processor 331 The user inputs of mouth 376 reception are processed.This process can include technology according to described in present disclosure and generate bag Include the packet of the user input order of reception.Once generating, transmission unit 333 can be processed to these packets So that network is transferred to radio source device on UIBC.

The processor 331 of Fig. 3 can include one or more of processor in extensive range, such as one or more numbers Word signal processor（DSP）, general purpose microprocessor, special IC（ASIC）, field programmable logic array（FPGA）, its Its equivalent integrated or discrete logic circuitry or its certain combination.The memorizer 332 of Fig. 3 can include various volatibility or non- Any one of volatile memory, including but not limited to：Such as Synchronous Dynamic Random Access Memory（SDRAM）Etc Random access memory（RAM）, read only memory（ROM）, nonvolatile RAM（NVRAM）, electric erasable can Program read-only memory（EEPROM）, flash memory etc..Memorizer 232 can be included for storing audio/video data and other The computer-readable recording medium of categorical data.Memorizer 332 can also store by processor 331 as execution present disclosure Described in a part for various technology and the instruction that executes and program code.

Fig. 4 shows example transmitter system 410 and the block diagram of receiver system 450, transmitter system 410 and receiver System 450 can be used for enterprising in communication channel 150 by the emittor/receiver 126 in Figure 1A and emittor/receiver 166 Row communication.At transmitter system 410, from data source 412 to transmitting（TX）Data processor 414 provides and is directed to multiple data flows Business datum.Each data flow can be sent on corresponding transmitting antenna.TX data processor 414 is based on for every number The business datum of each data flow is formatted, encodes and interweaves according to flowing selected specific encoding scheme.

OFDM can be used（OFDM）The encoded data of each data flow and pilot data are entered by technology Row multiplexing.Various other wireless communication technologys, including but not limited to time division multiple acess can also be used（TDMA）, frequency division multiple access （FDMA）, CDMA（CDMA）Or the combination in any of OFDM, FDMA, TDMA and/or CDMA.

With Fig. 4 as one man, the known data patterns that pilot data is typically processed in a known way, and can receive It is used at machine system estimating channel response.It is then possible to based on the certain modulation schemes selecting for each data flow（For example, Binary phase shift keying（BPSK）, QPSK（QPSK）, M-PSK or M-QAM（Quadrature amplitude modulation）, wherein M is permissible It is 2 power）The encoded data of multiplexed pilot data stream is modulated（For example, symbol mapping）To provide tune Symbol processed.The data rate of each data flow, coding and modulation can be determined by the instruction being executed by processor 430, its Middle processor 430 can be coupled with memorizer 432.

Then, the modulation symbol of data flow is supplied to TX MIMO processor 420, TX MIMO processor 420 can be right This modulation symbol is further processed（For example, for OFDM）.Then, TX MIMO processor 420 can be to N_TIndividual transmitter （TMTR）422a to 422t provides N_TIndividual stream of modulation symbols.In some aspects, TX MIMO processor 420 is by beam-forming weights The symbol being applied to data flow and the antenna sending this symbol.

Each transmitter 422 can receive and process corresponding symbol stream to provide one or more analogue signals, and Adjust further（For example, amplify, filter and up-conversion）This analogue signal is suitable for the warp of transmission on mimo channel to provide The signal of modulation.Then, respectively from N_TIndividual antenna 424a to 424t sends the N of spontaneous emission machine 422a to 422t_TIndividual modulated Signal.

At receiver system 450, by N_RIndividual antenna 452a to 452r receiving the modulated signal being sent, and And the receipt signal from each antenna 452 is supplied to corresponding receiver（RCVR）454a to 454r.Receiver 454 is adjusted （For example, filtering, amplification and down coversion）Respective receipt signal, is digitized to adjusted signal to provide sampling, and And process this sampling further to provide corresponding " reception " symbol stream.

Then, receive（RX）Data processor 460 can receive and be based on specific receiver treatment technology and process From N_RThe N of individual receiver 454_RThe symbol stream of individual reception, to provide N_TIndividual " detection " symbol stream.Then, RX data processor 460 can be demodulated, deinterleave and decode to the symbol stream that each detects, to recover the business datum of data flow.By RX number Hold with the TX MIMO processor 420 by transmitter system 410 and TX data processor 414 according to the process that processor 460 is carried out The process of row is complementary.

The processor 470 that can couple with memorizer 472 periodically determines used which pre-coding matrix.Reverse link Message can include the various types of information relevant with the data flow of communication link and/or reception.Then, reverse link message Processed by TX data processor 438（TX data processor 438 also receives the business number of multiple data flows from data source 436 According to）, it is modulated by manipulator 480, is adjusted by transmitter 454a to 454r, and be sent back to transmitter system 410.

At transmitter system 410, the modulated signal from receiver system 450 is received by antenna 424, by receiving Machine 422 is adjusted, and is demodulated by demodulator 440, and is processed by RX data processor 442, to extract by receiving The reverse link message that machine system 450 sends.Then, processor 430 determines which pre-coding matrix to determine wave beam becomes using Shape weight, is then processed to the message extracted.

Fig. 5 A is that the example being shown in the part as capability negotiation session between source device 520 and host device 560 disappears The block diagram of breath transmission sequence.Capability negotiation can be set up as the larger communication session between source device 520 and host device 560 Process a part of carrying out.It is, for example possible to use Wi-Fi directly or based on TDLS connectivity standard setting up this meeting Words.Setting up Wi-Fi directly or after TDLS session, host device 560 can be initiated to be connected with the TCP of source device 520.As building The part that vertical TCP connects, can set up operation real-time streaming protocol（RTSP）Control port, with to source device 520 Communication session and host device 560 between is managed.

Source device 520 generally can be to carry out with the identical mode described by the source device 120 above for Figure 1A Operation, and host device 560 generally can be to carry out with the identical mode described by the host device 160 above for Figure 1A Operation.After source device 520 and host device 560 set up connectivity, source device 520 and host device 560 can determine for it The parameter sets of subsequent communications session, the part exchanging as capability negotiation.

Source device 520 and host device 560 can be by a series of message come the abilities of consulting.These message can be for example real When streaming protocols（RTSP）Message.In any stage consulted, the recipient of RTSP request message can use except The RTSP also including RTSP state code outside RTSP OK responds to be responded, in which case it is possible to use different Parameter sets are retrying message exchange, or can terminate capability negotiation session.

Source device 520 can send first message to host device 560（RTSP selects request message）, to determine host device The RTSP method set of 560 supports.After receiving first message from source device 520, host device 560 can use the second message （RTSP Response to selection message）To be responded, this second message lists the RTSP method supported by place 560.This second message RTSPOK state code can also be included.

After sending the second message to source device 520, host device 560 can send the 3rd message（RTSP selects request Message）To determine the RTSP method set that source device 520 is supported.After receiving the 3rd message from host device 560, source device 520 can use the 4th message（RTSP Response to selection message）To be responded, the 4th message lists by source device 520 The RTSP method held.4th message can also include RTSP OK state code.

After sending the 4th message, source device 520 can send the 5th message（RTSP acquisition _ parameter request message）Come The list of the ability interested to specified source device 520.Host device 560 can use the 6th message（RTSP acquisition _ parameter responds Message）To be responded.6th message can comprise RTSP state code.If RTSP state code is OK, the 6th Message can also include the response parameter for the parameter supported by host device 560 specified in the 5th message.Host device 560 can ignore the parameter that in the 5th message, host device 560 is not supported.

Based on the 6th message, source 520 can determine the optimized parameter set for communication session, and can be to host device 560 transmission the 7th message（RTSP setting _ parameter request message）.7th message can comprise will set in source device 520 and place The parameter sets using during communication session between standby 560.7th message can include description and will set up request in RTSP Used in universal resource identifier（URI）Wfd-presentation-url to set up this communication session.wfd- Presentation-url specifies the URI that host device 560 can use during session establishment exchanges for message later.Should The value of wfd-url0 and wfd-url1 specifying in parameter can with the wfd-client-rtp-ports in the 7th message in Rtp-port0 with rtp-port1 value is corresponding.In this case, what RTP was commonly referred to as can running on UDP is real-time Agreement.

After receiving the 7th message, host device 560 can be carried out using the 8th message with RTSP state code Response, whether the instruction of this RTSP state code is successful according to the setting to parameter specified in the 7th message.Institute is mentioned above , in different sessions, role or source device and host device can invert or change.In some cases, set up communication meeting The order of the message of words can be defined as the source equipment being operated and the equipment being defined as Su Jinhang operation.

Fig. 5 B is to be shown in the another of the part as capability negotiation session between source device 560 and host device 520 to show The block diagram of example message transmission sequence.The message transmission sequence of Fig. 5 B aim to provide above for Fig. 5 A description transmission sequence more Detailed view.In figure 5b, message " response of 1b. acquisition _ parameter " shows the input classification that mark is supported（For example, lead to With and HIDC）List and the example of the message of multiple lists of input type supported.The row of the input classification supported The input classification that each of table is supported has the associated type supported（For example, generic_cap_list and hidc_cap_list）List.In figure 5b, message " 2a. setting _ parameter request " is the supported input classification of mark（Example As general and HIDC）Second list and the example of the second message of multiple second lists of type supported.Supported The input classification that each of second list of input classification is supported has the associated type supported（For example, Generic_cap_list and hidc_cap_list）Second list.Message " response of 1b. acquisition _ parameter " identifies by host device Input classification and input type that 560 are supported.Message " 2a. setting _ parameter request " mark by source device 520 supported defeated Enter classification and input type, but it can not be the comprehensive of all input classifications supported by source device 520 and input type List.But, message " 2a. setting _ parameter request " can only identify and be designated in message " response of 1b. acquisition _ parameter " Those the input classifications supported by host device 560 and input type.By this way, in message " 2a. setting _ parameter request " The input classification of middle mark and input type may be constructed in message " response of 1b. acquisition _ parameter " the input classification of mark and The subset of input type.

Fig. 6 is the concept illustrating to be generated and sent to an example of packet of source device by host device Figure.With reference to Figure 1A, each side of data packet 600 will be explained, but the technology being discussed can apply to other type Source/place system.Packet 600 can include data packet header 610, data packet header 610 followed by payload Data 650.Payload data 650 can comprise additionally in one or more payload headers（For example, payload header 630）.For example, it is possible to packet 600 is sent to source device 120 from the host device 160 of Figure 1A, so that host device 160 User can control the audio/video data being sent by source device 120.In this case, payload data 650 is permissible Including the user input data receiving at host device 160.For example, payload data 650 can identify one or more use Family order.Host device 160 can receive this one or more user command, and can generate data based on the order being received Packet header 610 and payload data 650.The content of the data packet header 610 based on packet 600, source device 120 Payload data 650 can be parsed, to identify the user input data receiving at host device 160.Being based on is having User input data included in effect load data 650, source device 120 can change in some way from source device 120 Deliver to the Voice ＆ Video data of host device 160.

As used in present disclosure, term " parsing " and " parsing " are commonly referred to as bit stream is carried out point Analysis is to extract the process of data from this bit stream.Once being extracted, for example, it is possible to be carried out to this data by source device 120 Reason.For example, extract data can include identifying how formatted the data in this bit stream is.As will be detailed below in more detail Description, data packet header 610 can define all known standardized format to both source device 120 and host device 160.So And, can by many possible in the way of one of mode payload data 650 is formatted.By to packet Header 610 is parsed, and source device 120 can determine how formatted effective load data 650 is, and thus, source Equipment 120 can parse to payload data 650, to extract one or more users from payload data 650 Input order.Motility is provided in terms of this different types of payload data that can be supported in-the Su Tongxin of source.As follows Face is explained in greater detail, and payload data 650 can also include of such as payload header 630 etc or many Individual payload header.In these cases, source device 120 can be parsed to data packet header 610 to determine effectively The form of payload header 630, and and then payload header 630 is parsed to determine the surplus of payload data 650 The form that remaining part is divided.

Figure 62 0 is the conceptual description with regard to can how to format to data packet header 610.In row 615 Digital 0-15 is intended to the bit position in mark data packet header 610, and is not intended to practically represent packet The information comprising in header 610.Data packet header 610 include version field 621, timestamp mark 622, reserved field 623, Input classification field 624, length field 625 and optional timestamp field 626.

In the example of fig. 6, version field 621 is 3 bit fields, its can indicate by host device 160 realize specific The version of communication protocol.Value in version field 621 can inform source device 120 residue how to data packet header 610 Part is parsed, and how payload data 650 is parsed.In the example of fig. 6, version field 621 is 3 ratios Special field, it is capable of the unique identifier for 8 kinds of different editions.In other examples, more or less of bit can With dedicated for version field 621.

In the example of fig. 6, timestamp mark（T）622 is 1 bit field, and its instruction is in data packet header 610 No existence time stabs field 626.Timestamp field 626 is 16 bit fields, and it comprises based on being generated and sent out by source device 120 Deliver to the timestamp of the multi-medium data of host device 160.For example, this timestamp can be that frame of video is being sent to host device Distributed to the sequence valve of this frame by source device 120 before 160.In the presence of timestamp mark 622 can for example be included for instruction Between stab field 626 " 1 ", and can include for instruction not existence time stab field 626 " 0 ".To packet report After 610 are parsed and determined existence time stamp field 626, source device 120 can be to including in timestamp field 626 Timestamp is processed.After data packet header 610 being parsed and is determined that existence time does not stab field 626, due to In data packet header 610, existence time does not stab field, and therefore source device 120 can parse to length field 625 Start afterwards payload data 650 is parsed.

If it does, timestamp field 626 can be included for mark when the user input obtaining payload data 650 The timestamp of the video data frame showing at wireless host device 160 during data.For example, in source device 120, this frame of video is sent out Before delivering to host device 160, this timestamp can be added to by this frame of video by source device 120.Therefore, source device 120 can To generate frame of video and to embed timestamp in the video data of this frame（For example as metadata）.Source device 120 can be to Host device 160 sends the frame of video with timestamp, and host device 160 can show this frame of video.When being shown by host device 160 When showing this frame of video, host device 160 can be from user's receive user order.When host device 160 generates packet with Xiang Yuanshe During standby 120 this user command of transmission, the frame that host device 160 can will be shown by host device 160 when receiving this user command Timestamp include in timestamp field 626.

After receiving the packet 600 with the timestamp field 626 being present in header, radio source device 120 The frame of video of the display at host device 160 when obtaining the user input data of payload data 650 can be identified, and can With the content based on the frame being identified by timestamp, user input data is processed.For example, if user input data is It is applied to the touch order of touch display or the click of mouse pointer, then source device 120 can determine when user is to display Application touch order or the content clicking on shown frame during mouse.In some cases, it can be possible to need frame content come right Payload data carries out suitable process.For example, be may rely on based on the user input of user's touch or click and work as The content just showing over the display when touching or clicking on.For example, touch or click can be corresponding with icon or menu option. In the case that the content of display changes, timestamp present in field 626 can be stabbed by source device 120 use time Match correct icon or menu option by touching or clicking on.

Additionally or alternatively, source device 120 can by the timestamp in timestamp field 626 be applied to currently present The timestamp of frame of video be compared.By the timestamp in timestamp field 626 and current time stamp are compared, source Equipment 120 can determine two-way time.This two-way time is generally with the moment from source device 120 transmission frame in source device 120 Elapsed time in the moment amount that place is received back to the user input based on this frame from host device 160 is corresponding.This two-way time can To provide the instruction to system delay to source device 120, and if this two-way time is more than threshold value, then answered in input order Under hypothesis for out-of-date display frame, source device 120 can ignore the user input number being included in payload data 650 According to.When this two-way time being less than threshold value, source device 120 can be processed to user input data, and in response to this User input data and the audio/video content that sends are adjusted.Threshold value can be programmable, and different types of sets Standby（Or different source-Su Zuhe）It is configurable to define the acceptable different threshold values for two-way time.

In the example of fig. 6, reserved field 623 is 8 bit fields, and it is not included by source device 120 to packet The information that header 610 and payload data 650 are used when being parsed.However,（As marked in version field 621 Know）The future version of specific protocol can use reserved field 623, and in this case, source device 120 can be using reservation Information in field 623 data packet header 610 to be parsed and/or payload data 650 is parsed.Retain Field 623 with reference to version field 621 potentially provide not to use form and feature be fundamentally changed In the case of, data packet format is extended and adds with the ability of feature.

In the example of fig. 6, input classification field 624 is 4 bit fields, and it is used for identifying in payload data 650 The input classification of the user input data being comprised.Host device 160 can be classified to user input data to determine input Classification.For example, user input data being carried out classifies can be based on the equipment receiving from it order or based on order The attribute of itself.The value of input classification field 624（May be in conjunction with the other information of data packet header 610）To source device How formatted 120 mark payload datas 650 are.Based on this formatting, source device 120 can be to payload number Parsed according to 650 to determine the user input receiving at host device 160.

In the example of fig. 6, because input classification field 624 is 4 bits, 16 kinds of different input classes can therefore be identified Not.A kind of such input classification can be universal input form, and the user input data of its instruction payload data 650 is Usually format using in the general information unit defined in the performed agreement of both source device 120 and host device 160 's.As will be described in greater detail below, universal input form can be using the user of permission host device 160 in application layer and source The general information element that equipment 120 interacts.

Another this input classification can be human interface device order（HIDC）Form, it indicates payload data 650 user input data is to be formatted based on the type of the input equipment for receiving this input data.Equipment The example of type include：Keyboard, mouse, touch input device, stick, photographing unit, gesture capture device（Such as it is based on and shine The input equipment of camera）And remotely control.The other types of input classification bag that can identify in input classification field 624 Include：Indicate payload data 650 in user input data be not derived from host device 160 forwarding pattern of the input or Include the voice command form of voice command specific to the form of operating system and instruction payload data 650.

Length field 625 can include 16 bit fields of the length for indicating packet 600.For example, it is possible to 8 Bit to indicate this length for unit.Because packet 600 is to be parsed with the word of 16 bits by source device 120, because This can be filled with to packet 600 to reach the integral multiple of 16 bits.Based on the length comprising in length field 625, Source device 120 can identify the end of payload data 650（The i.e. end of packet 600）Divide with new follow-up data The beginning of group.

The all size of the field providing in the example of Fig. 6 is only intended to be explanatory, and is intended to use and Fig. 6 Shown in varying number bit realizing these fields.It is also assumed that, data packet header 610 is permissible Including the field more less than all fields discussed above or can be using the extra field that is not discussed above.When So, in terms of the actual format for each data field in packet, the technology of present disclosure can be flexible.

After data packet header 610 being parsed to determine the formatting to payload data 650, source device 120 can be parsed to payload data 650 to determine the user input order comprising in payload data 650.Have Effect load data 650 can have the payload header of their own（Payload header 630）, it indicates payload data 650 content.By this way, source device 120 can based on to the parsing of data packet header 610 come to payload header 630 are parsed, and the parsing that may then based on to payload header 630 payload data 650 is solved Analysis.

For example, exist in payload data 650 if the input classification field 624 of data packet header 610 indicates Universal input, then payload data 650 can have universal input form.Therefore, source device 120 can be general defeated according to this Entry format parses to payload data 650.As a part for universal input form, payload data 650 is permissible Including a series of one or more incoming events, wherein each incoming event has the incoming event header of their own.Below Table 1 identify can be in the input field that include of header.

Form 1

Field	Size（Eight bit byte）	Value
			General purpose I EID	1	It is shown in Table 2
Length	2	Length in units of eight bit byte for the field below
			Description	Variable	The details of user input.See each table

Universal input event（IE）Mark（ID）Field identification is used for identifying the universal input event identifier of input type.Example As general purpose I E id field with length for 1 eight bit byte, and can include the mark selecting from table 2 below.As In this example, if general purpose I E id field is 8 bits, 256 kinds of different types of inputs can be identified（It is designated 0- 255）Although being not all of all certain input type needing to be associated of 256 kinds of marks.Some in this 256 can retain So that the future version of any agreement for being realized by host device 160 and source device 120 in the future.In table 2, for example, general IE ID 9-255 does not have associated input type, but can distribute input type to it in the future.

Length field in incoming event header identifies the length of description field, and description field includes describing user input Information element.Formatting to description field can be depending on the type of the input of mark in general purpose I E id field.Therefore, Source device 120 can be parsed to the content of description field based on the input type of mark in general purpose I E id field.It is based on The length field of incoming event header, source device 120 can determine the end of one of effective load data 650 incoming event Beginning with new incoming event.Will be explained in more detail as following, can be by a user in payload data 650 Command description is one or more incoming events.

Table 2 provides the example of input type, and each input type has and can be used for identifying the corresponding of this input type General purpose I E ID.

Form 2

General purpose I E ID	Input type
		0	Left mouse down/touch is pressed
1	Left mouse-up/touch is lifted

2	Mouse moves/touches and moves
		3	Button is pressed
4	Button lifts
		5	Scaling
6	Vertical scrolling
		7	Horizontal rolling
8	Rotation
		9-255	Retain

The description field being associated from each input type can have different forms.For example, left mouse down/touch Press event, event is lifted in left mouse-up/touch and mouse mobile/touch movement description field can include following Table 3 in mark information element, although can also be using other forms in other examples.

Form 3

The quantity of pointer can identify the number of times of the touch being associated with incoming event or click.Each pointer is permissible There is unique pointer ID.For example, if multi-touch event includes three fingers and touches, incoming event can have three fingers Pin, wherein each pointer have unique pointer ID.Each pointer（That is, each finger touches）Can have and touch point Corresponding x coordinate and y-coordinate.

Unique user command description can be a series of incoming event.For example, if it is to close application that three fingers are gently swept Order, then three can be referred to gently sweep the touch being described as thering is three pointers and press event, tool in payload data 650 Event is lifted in the touch moving event having three pointers and the touch with three pointers.Three pointers touching event of pressing can With three pointer identical pointer ID having with touch moving event and touch event of lifting.Source device 120 can by this three The combination interpretation of individual incoming event is that three fingers are gently swept.

For example, key press event or button lift the description field of event and can include the letter of mark in table 4 below Breath element.

Form 4

For example, the description field of scaling event can include the information element of mark in table 5 below.

Form 5

For example, the description field of horizontal rolling event or vertical scrolling event can include the letter of mark in table 6 below Breath element.

Form 6

Above example shows that some that for universal input classification, payload data can be formatted are shown Example property mode.If the input classification field 624 of data packet header 610 indicates different input classifications（The use of such as forwarding Family inputs）, then payload data 650 can have different pattern of the inputs.In the situation of the user input forwarding, host device 160 can be from third party device receiving user's input data, and in the case of this user input data not being explained This input is forwarded to source device 120.Thus, source device 120 can be according to the user input format forwarding to payload number Parsed according to 650.For example, the payload header 630 of payload data 650 can include obtaining use for mark from it The field of the third party device of family input.For example, this field can include the Internet Protocol of third party device（IP）Address, MAC Address, domain name or some this class identifiers other.Source device 120 can the identifier based on third party device carry to effective The remainder of lotus data is parsed.

Host device 160 can consult ability via a series of message with third party device.Then, as capability negotiation procedure A part, host device 160 can to source device 120 send third party device unique identifier, build as with source device 120 A part for vertical communication session.Or, host device 160 can send the information of description third party device to source device 120, and Based on this information, source device 120 can determine the unique identifier of third party device.For example, the information of third party device is described The information for identifying third party device and/or the information for identifying the ability of third party device can be included.No matter this is only One identifier is still determined by host device 160 by source device 120, obtains from third party device when host device 160 sends to have During the packet of user input obtaining, host device 160 can be included this unique identifier in packet（For example, wrap Include in payload header）, so that source device 120 is capable of identify that the source of this user input.

If the input classification field 624 of data packet header 610 indicates another different input classification, such as voice life Order, then payload data 650 can have another different input type.For voice command, payload data 650 can To include encoded audio frequency.Can via a series of message between source device 120 and host device 160 to for voice order The audio frequency of order is encoded and the codec that decoded is held consultation.In order to send voice command, timestamp field 626 can be wrapped Include speech sample time value.In such a case, it is possible to timestamp mark 622 is set to indicate that existence time is stabbed, rather than Timestamp as described above, timestamp field 626 can include the encoded audio frequency for payload data 650 Speech sample time value.

In some instances, as set forth above, it is possible to voice command is transmitted as generic command, in this situation Under, input classification field 626 can be set to identify in generic command form, and the general purpose I E ID that can will retain Distribute to voice command for one.If be transmitted voice command as generic command, speech sample rate may reside in In the timestamp field 626 of data packet header 610 or may reside in payload data 650.

For the voice command data of capture, can in many ways speech data be packaged.It is, for example possible to use RTP voice command data is packaged, and RTP can provide PT Payload Type to identify codec and timestamp, its In this timestamp be used for identifying sample rate.In the case of with or without optional timestamp, it is possible to use be described above Common user pattern of the input RTP data is packaged.Host device 160 can be sent to source device 120 using TCP/IP Carry the universal input data of voice command data.

As previously discussed, when using coordinate as packet（Such as packet 600）A part include in example As when in payload data 650, this coordinate can with the coordinate based on the resolution scaling consulted, display window coordinate, return One change coordinate or the coordinate associated with place display are corresponding.In some cases, extra information can be included In packet or individually it is transmitted, to be used for the coordinate receiving in packet is returned by source device One change.

Do not consider the input classification of specific data packet, data packet header can be application layer packet header, and can So that packet is sent by TCP/IP.TCP/IP can enable host device 160 and source device 120 in the feelings of packet loss Retransmission technique is executed under condition.Packet can be sent to source device 120 to control the sound of source device 120 from host device 160 Frequency evidence or video data, or for other purposes（Such as control the application operating on source device 120）.

Fig. 7 A is the flow chart of the exemplary method consulting ability between host device and source device.Shown exemplary method Can be by host device 160（Figure 1A）Or 360（Fig. 3）To execute.In some instances, computer-readable recording medium（For example, deposit Reservoir 332）Can be with store instruction, module or algorithm, when this instruction, module or algorithm are performed so that one or more place Reason device（For example, processor 331）Execute shown in one or more of flow chart described herein flow chart One or more of step step.

The method of Fig. 7 A includes host device 160 and receives first message from source device 120（701）.For example, this message can be wrapped Include acquisition parameter request.In response to this first message, host device 160 can send the second message to source device 120（703）.Example As this second message can include obtaining parameter response, the first row of the input classification that this acquisition parameter response identification is supported Table and multiple first lists of the type supported.Wherein, each of first list of input classification supported is propped up The input classification held has the first list of the associated type supported.For example, the input classification supported can with Corresponding in the input classification field 624 identical classification of Fig. 6.Table 2 above represents for specific input classification（Show at this It is universal input in example）The type supported an example.Host device 160 can receive the 3rd message from source device 120 （705）.For example, the 3rd message can include arrange parameter request, and wherein, this arrange parameter request mark is for the end of communication Multiple second lists of mouth, the second list of input classification supported and the type supported, wherein, that is supported is defeated Enter the second list that the input classification that each of second list of classification supported has the associated type supported, And the type that each of second list is supported includes the subset of the type in first list.Host device 160 can be to Source device 120 sends the 4th message（707）.For example, the 4th message can be included for confirming to have enabled in second list Type arrange parameter response.Host device 160 can receive the 5th message from source device 120（709）.For example, the 5th disappears Breath can include indicating the second arrange parameter request having enabled the communication channel between source device 120 and host device 160.Example As this communication channel can include user input Return Channel（UIBC）.Host device 160 can send the 6th to source device 120 Message（711）.For example, the 6th message can include validating that host device 160 to the reception that the second arrange parameter is asked second Arrange parameter responds.

Fig. 7 B is the flow chart of the exemplary method consulting ability between host device and source device.Shown exemplary method Can be by source device 120（Figure 1A）Or 220（Fig. 2）To execute.In some instances, computer-readable recording medium（For example, deposit Reservoir 232）Can be with store instruction, module or algorithm, when this instruction, module or algorithm performs so that one or more process Device（For example, processor 231）Execute one or more of shown step in this flow chart step.

The method of Fig. 7 B includes：Source device 120 sends first message to host device 160（702）.For example, this first message Can include obtaining parameter request.Source device 120 can receive the second message from host device 160（704）.For example, this second disappears Breath can include obtaining parameter response, the first list of input classification that this acquisition parameter response identification is supported with supported Multiple first lists of type, wherein, the input classification that each of first list of input classification supported is supported There is the first list of the associated type supported.Source device 120 can send the 3rd message to host device 160（706）. For example, the 3rd message can include arrange parameter request, this arrange parameter request mark for communication port, supported The input second list of classification and multiple second lists of the type supported, wherein, supported the second of input classification The input classification that each of list is supported has the second list of the associated type supported, and second list Each of the type supported include the subset of the type in first list.Source device 120 can receive from host device 160 4th message（708）.For example, the 4th message can include the setting ginseng for confirming to have enabled the type in second list Number response.Source device 120 can send the 5th message to host device 160（710）.For example, the 5th message can include indicating Enable the second arrange parameter request of the communication channel between source device 120 and host device 160.For example, this communication channel User input Return Channel can be included（UIBC）.Source device 120 can receive the 6th message from host device 160（712）.Example As the 6th message can include validating that the second arrange parameter response to the reception that the second arrange parameter is asked for the host device 160.

Fig. 8 A is according to present disclosure, sends the example side of user input data from wireless host device to radio source device The flow chart of method.Shown exemplary method can be by host device 160（Figure 1A）Or 360（Fig. 3）To execute.In some examples In, computer-readable recording medium（For example, memorizer 232）Can with store instruction, module or algorithm, when this instruction, module or So that one or more processors during algorithm performs（For example, processor 231）Execute in the shown step in this flow chart One or more steps.

The method of Fig. 8 A is included in wireless host device（Such as wireless host device 160）Place obtains user input data（801）. Can be by the user input assembly of wireless host device 160（Such as, for example, the user input shown in combining wireless host device 360 Interface 376）To obtain user input data.Additionally, host device 160 can by user input data be categorized as example general, Forward or specific to operating system.Then, host device 160 can generate data packet header based on user input data （803）.This data packet header can be application layer packet header.In addition to other fields, this data packet header also may be used To include the field for identifying the input classification corresponding with user input data.For example, input classification can include general Pattern of the input or human interface device order.Host device 160 can also generate packet（805）, wherein, this packet bag Include generated data packet header and payload data.In one example, payload data can include being received User input data, and one or more user commands can be identified.Then, host device 160 can be to radio source device （For example, the 220 of the source device 120 of Figure 1A or Fig. 2）Send generated packet（807）.Host device 160 can include permitting Permitted to transmit the assembly of packet, for example, included the transmission unit 333 shown in Fig. 3 and radio modem 334.Host device 160 can transmit packet by TCP/IP.

Fig. 8 B is according to present disclosure, from the example of wireless host device receiving user's input data at radio source device The flow chart of method.Shown exemplary method can be by source device 120（Figure 1A）Or 220（Fig. 2）To execute.In some examples In, computer-readable recording medium（For example, memorizer 232）Can with store instruction, module or algorithm, when this instruction, module or So that one or more processors during algorithm performs（For example, processor 231）Execute in the shown step in this flow chart One or more steps.

The method of Fig. 8 B includes receiving data packet（802）, wherein, among other things, this packet is permissible Including data packet header and payload data.Payload data can include such as user input data.Source device 120 Can include allowing the communication component of transmission packet, it is included referring for example to the transmission unit 233 shown in Fig. 2 and wireless tune Modulator-demodulator 234.Then, source device 120 can parse to including the data packet header in packet（804）, To determine the input classification being associated with the user input data being included in this payload data.Source device 120 can be with base In determined by input classification this payload data is processed（806）.The packet of reference picture 8A and 8B description is led to The form with reference to the packet described by Fig. 6 often can be taken, and can be used for controlling the audio/video number at source device According to and application.

Fig. 9 A is according to present disclosure, sends the example side of user input data from wireless host device to radio source device The flow chart of method.Shown exemplary method can be by host device 160（Figure 1A）Or 360（Fig. 3）To execute.In some examples In, computer-readable recording medium（For example, memorizer 332）Can with store instruction, module or algorithm, when this instruction, module or So that one or more processors during algorithm performs（For example, processor 331）Execute in the shown step in this flow chart One or more steps.

The method of Fig. 9 A is included in wireless host device（Such as wireless host device 160）Place obtains user input data（901）. Can be by the user input assembly of wireless host device 160（Such as, for example, referring to the user input interface 376 shown in Fig. 3）Come Obtain user input data.Then, host device 160 can generate payload data（903）, wherein this payload data can To describe user input data.In one example, payload data can include received user input data and One or more user commands can be identified.Host device 160 can also generate packet（905）, wherein, this packet bag Include data packet header and the payload data being generated.Then, host device 160 can be to radio source device（For example, Figure 1A Source device 120 or Fig. 2 220）Send generated packet（907）.Host device 160 can include allowing transmission data The assembly of packet, such as, for example, transmission unit 333 and radio modem 334.Can be by TCP/IP by packet It is sent to radio source device.

Fig. 9 B is according to present disclosure, from the example of wireless host device receiving user's input data at radio source device The flow chart of method.Shown exemplary method can be by source device 120（Figure 1A）Or 220（Fig. 2）To execute.In some examples In, computer-readable recording medium（For example, memorizer 232）Can with store instruction, module or algorithm, when this instruction, module or So that one or more processors during algorithm performs（For example, processor 231）Execute in the shown step in this flow chart One or more steps.

The method of Fig. 9 B includes being grouped from host device 360 receiving data（902）, wherein, among other things, this number Data packet header and payload data can be included according to packet.In one example, payload data can include example As described the data of the details of user input（Such as input type value）.Source device 120 can include allowing transmission packet Communication component, including referring for example to transmission unit 233 illustrated in fig. 2 and radio modem 234.Then, source device 120 can parse to this packet（904）, to determine the input class in the input type field in payload data Offset.Source device 120 can based on determined by input type value the data of details of description user input is processed （906）.The packet of reference picture 9A and 9B description generally can take the form of the packet with reference to Fig. 6 description.

Figure 10 A is according to present disclosure, sends the example of user input data from wireless host device to radio source device The flow chart of method.Shown exemplary method can be by host device 160（Figure 1A）Or 360（Fig. 3）To execute.In some examples In, computer-readable recording medium（For example, memorizer 332）Can with store instruction, module or algorithm, when this instruction, module or So that one or more processors during algorithm performs（For example, processor 331）Execute in the shown step in this flow chart One or more steps.

The method of Figure 10 A is included in wireless host device（Such as wireless host device 160）Place obtains user input data （1001）.Can be by the user input assembly of wireless host device 160（Such as, connect for example, referring to the user input shown in Fig. 3 Mouth 376）To obtain user input data.Then, host device 160 can generate data packet header based on user input （1003）.In addition to other fields, whether this data packet header can include depositing in data packet header for instruction Timestamp mark in timestamp field（For example, 1 bit field）.For example, in the presence of this timestamp mark can include instruction Between stab " 1 " of field, and can include indicating that not existence time stabs " 0 " of field.For example, this timestamp field can be 16 Bit field, it comprises to be generated and be added to before being transmitted by source device 120 timestamp of video data.Host device 160 Packet can also be generated（1005）, wherein, this packet includes generated data packet header and payload number According to.In one example, payload data can include received user input data and can identify one or many Individual user command.Then, host device 160 can be to radio source device（For example, the 220 of the source device 120 of Figure 1A or Fig. 2）Send The packet being generated（1007）.Host device 160 can include allow transmission packet assembly, it include referring for example to Transmission unit 333 illustrated in fig. 3 and radio modem 334.Can be sent the packets to wirelessly by TCP/IP Source device.

Figure 10 B is according to present disclosure, showing from wireless host device receiving user's input data at radio source device The flow chart of example method.Shown exemplary method can be by source device 120（Figure 1A）Or 220（Fig. 2）To execute.Show at some In example, computer-readable recording medium（For example, memorizer 232）Can be with store instruction, module or algorithm, when this instruction, module Or so that one or more processors during algorithm performs（For example, processor 231）Execute the shown step in this flow chart One or more of step.

The method of Figure 10 B includes being grouped from wireless host device 160 receiving data（1002）, wherein, except other side it Outward, this packet can include data packet header and payload data.For example, payload data can include user Input data.Source device 120 can include allowing the communication component of transmission packet, and it is included referring for example to shown by Fig. 2 Transmission unit 233 and radio modem 234.Then, source device 120 can divide to the data that this packet includes Group header is parsed（1004）.Source device 120 can determine and whether there is timestamp field in this data packet header （1006）.In one example, source device 120 can be entered based on the timestamp value of statistical indicant including in this data packet header This determination of row.If this data packet header includes timestamp field, source device 120 can be based in this timestamp field Timestamp is processed to payload data（1008）.The packet of reference picture 10A and 10B description generally can take ginseng According to the form of the packet of Fig. 6 description, and can be used for controlling the audio/video data at source device.

Figure 11 A is according to present disclosure, sends the example of user input data from wireless host device to radio source device The flow chart of method.Shown exemplary method can be by host device 160（Figure 1A）Or 360（Fig. 3）To execute.In some examples In, computer-readable recording medium（For example, memorizer 332）Can with store instruction, module or algorithm, when this instruction, module or So that one or more processors during algorithm performs（For example, processor 331）Execute in the shown step in this flow chart One or more steps.

The method of Figure 11 A is included in wireless host device（Such as wireless host device 160）Place obtains user input data （1101）.Can be by the user input assembly of wireless host device 160（Such as, connect for example, referring to the user input shown in Fig. 3 Mouth 376）To obtain this user input data.Then, host device 160 can generate data packet header based on user input （1103）.In addition to other fields, this data packet header can include timestamp field.For example, this timestamp field can To include 16 bit fields, this 16 bit field comprises to set based on being generated and transmitted to wireless place by radio source device 120 The timestamp of standby 160 multi-medium data.Can be before being sent to wireless host device, during by radio source device 120 by this Between stamp be added to video data frame.For example, this timestamp field can identify and when capturing user input data, in wireless place The timestamp that at equipment 160, the video data frame of display is associated.Host device 160 can also generate packet（1105）, its In, this packet includes generated data packet header and payload data.In one example, payload data Received user input data can be included and one or more user commands can be identified.Then, host device 160 is permissible To radio source device（For example, the 220 of the source device 120 of Figure 1A or Fig. 2）Send generated packet（1107）.Host device 160 can include allowing the assemblies of transmission packet, and it is included referring for example to transmission unit illustrated in fig. 3 333 and wireless Modem 334.Radio source device can be sent the packets to by TCP/IP.

Figure 11 B is according to present disclosure, showing from wireless host device receiving user's input data at radio source device The flow chart of example method.Shown exemplary method can be by source device 120（Figure 1A）Or 220（Fig. 2）To execute.Show at some In example, computer-readable recording medium（For example, memorizer 232）Can be with store instruction, module or algorithm, when this instruction, module Or so that one or more processors during algorithm performs（For example, processor 231）Execute the shown step in this flow chart One or more of step.

The method of Figure 11 B is included from wireless host device（Such as wireless host device 160）Receiving data is grouped（1102）, its In, among other things, this packet can include data packet header and payload data.For example, effectively carry Lotus data can include user input data.Source device 120 can include allowing the communication component of transmission packet, and it includes Referring for example to transmission unit 233 illustrated in fig. 2 and radio modem 234.Then, source device 120 can be with identification data Timestamp field in packet header（1104）.Source device 120 can be based on the timestamp in timestamp field to payload Data is processed（1106）.As the part that payload data is processed, source device 120 can be based on timestamp To identify the video data frame of the display at wireless host device when obtaining this user input data, and the content based on this frame To explain this payload data.As part payload data being processed based on this timestamp, source device 120 The current time stamp of this timestamp and the current video frame being sent by source device 120 can be compared, and can respond Time difference between this timestamp and current time stamp to execute the user input described in payload data less than threshold value Order, or be more than threshold value in response to the time difference between this timestamp and current time stamp and do not execute in payload data The user input order of description.The packet of reference picture 11A and 11B description generally can take the data with reference to Fig. 6 description The form of packet, and can be used for controlling the audio/video data at source device.

Figure 12 A is according to present disclosure, sends the example of user input data from wireless host device to radio source device The flow chart of method.Shown exemplary method can be by host device 160（Figure 1A）Or 360（Fig. 3）To execute.In some examples In, computer-readable recording medium（For example, memorizer 332）Can with store instruction, module or algorithm, when this instruction, module or So that one or more processors during algorithm performs（For example, processor 331）Execute in the shown step in this flow chart One or more steps.

The method of Figure 12 A is included in wireless host device（Such as wireless host device 160）Place obtains user input data （1201）.In one example, this user input data can be voice command data, can be by wireless host device 160 User input assembly（Such as, for example, the voice command recognition module that the user input interface 376 in Fig. 3 includes）To obtain This voice command data.Host device 160 can generate data packet header based on user input（1203）.Host device 160 is also Payload data can be generated（1205）, wherein this payload data can include voice command data.In an example In, payload data can also include received user input data and can identify one or more user commands. Host device 160 can also generate packet（1207）, wherein, this packet includes generated data packet header and has Effect load data.Then, host device 160 can be to radio source device（For example, the 220 of the source device 120 of Figure 1A or Fig. 2）Send The packet being generated（1209）.Host device 160 can include allow transmission packet assembly, it include referring for example to Transmission unit 333 illustrated in fig. 3 and radio modem 334.Can be sent the packets to wirelessly by TCP/IP Source device.

Figure 12 B is according to present disclosure, showing from wireless host device receiving user's input data at radio source device The flow chart of example method.Shown exemplary method can be by source device 120（Figure 1A）Or 220（Fig. 2）To execute.Show at some In example, computer-readable recording medium（For example, memorizer 232）Can be with store instruction, module or algorithm, when this instruction, module Or so that one or more processors during algorithm performs（For example, processor 231）Execute the shown step in this flow chart One or more of step.

The method of Figure 12 B includes receiving data packet（1202）, wherein, among other things, this packet can To include data packet header and payload data.For example, payload data can include such as voice command data it The user input data of class.Source device 120 can include allowing the communication component of transmission packet, and it is included referring for example to figure Transmission unit 233 shown by 2 and radio modem 234.Then, source device 120 can include to this packet Payload data parsed（1204）, to determine whether this payload data includes voice command data.Reference picture The packet of 12A and 12B description generally can take the form of the packet with reference to Fig. 6 description, and can be used for controlling Audio/video data at source device processed.

Figure 13 A is according to present disclosure, sends the example of user input data from wireless host device to radio source device The flow chart of method.Shown exemplary method can be by host device 160（Figure 1A）Or 360（Fig. 3）To execute.In some examples In, computer-readable recording medium（For example, memorizer 332）Can with store instruction, module or algorithm, when this instruction, module or So that one or more processors during algorithm performs（For example, processor 331）Execute in the shown step in this flow chart One or more steps.

The method of Figure 13 A is included in wireless host device（Such as wireless host device 160）Place obtains user input data （1301）.In one example, this user input data can be multi-touch gesture, and it can be by wireless host device 160 User input assembly（Such as, for example, the user input interface 376 in UI 167 or Fig. 3）To obtain.In one example, This multi-touch gesture can include the first touch input and the second touch input.Host device 160 can based on user input Lai Generate data packet header（1303）.Host device 160 can also generate payload data（1305）, wherein this payload number According to the user input data for the first touch input event being associated with the first pointer mark, and will be for second The user input data of touch input event is associated with the second pointer mark.Host device 160 can also generate packet （1307）, wherein, this packet includes generated data packet header and payload data.Then, host device 160 can With to radio source device（For example, the 220 of the source device 120 of Figure 1A or Fig. 2）Send generated packet（1309）.Place sets Standby 160 can include allowing the assembly of transmission packet, and it is included referring for example to transmission unit 333 illustrated in fig. 3 and nothing Line modem 334.Radio source device can be sent the packets to by TCP/IP.

Figure 13 B is according to present disclosure, showing from wireless host device receiving user's input data at radio source device The flow chart of example method.Shown exemplary method can be by source device 120（Figure 1A）Or 220（Fig. 2）To execute.Show at some In example, computer-readable recording medium（For example, memorizer 232）Can be with store instruction, module or algorithm, when this instruction, module Or so that one or more processors during algorithm performs（For example, processor 231）Execute the shown step in this flow chart One or more of step.

The method of Figure 13 B includes receiving data packet（1302）, wherein, among other things, this packet can To include data packet header and payload data.For example, payload data can include such as multi-touch gesture it The user input data of class.Source device 120 can include allowing the communication component of transmission packet, and it is included referring for example to figure Transmission unit 233 shown by 2 and radio modem 234.Then, source device 120 can include to this packet Payload data parsed（1304）, to identify the user input data that this payload data includes.At one In example, the data being identified can include the user input number for the first touch input event with the first pointer mark According to this and there is the user input data for the second touch input event of the second pointer mark.Then, source device 120 is permissible User input data for the first touch input event and the user input data for the second touch input event are explained For multi-touch gesture（1306）.The packet of reference picture 13A and 13B description generally can take the number with reference to Fig. 6 description According to the form of packet, and can be used for controlling the audio/video data at source device.

Figure 14 A is according to present disclosure, sends the example of user input data from wireless host device to radio source device The flow chart of method.Shown exemplary method can be by host device 160（Figure 1A）Or 360（Fig. 3）To execute.In some examples In, computer-readable recording medium（For example, memorizer 332）Can with store instruction, module or algorithm, when this instruction, module or So that one or more processors during algorithm performs（For example, processor 331）Execute in the shown step in this flow chart One or more steps.

The method of Figure 14 A includes obtaining user input data from external equipment at wireless host device 360（1401）.One In individual example, this external equipment can be attached to the third party device of this host device.Host device 160 can be based on user input To generate data packet header（1403）.In one example, user input data can be designated by this data packet header The user input data forwarding.Host device 160 can also generate payload data（1405）, wherein, this payload data User input data can be included.Host device 160 can also generate packet（1407）, wherein, this packet includes institute The data packet header generating and payload data.Then, host device 160 can be to radio source device（For example, the source of Figure 1A Equipment 120 or the 220 of Fig. 2）Send generated packet（1409）.Host device 160 can include allowing transmission data to divide The assembly of group, it is included referring for example to transmission unit 333 illustrated in fig. 3 and radio modem 334.TCP/ can be passed through IP sends the packets to radio source device.

Figure 14 B is according to present disclosure, showing from wireless host device receiving user's input data at radio source device The flow chart of example method.Shown exemplary method can be by source device 120（Figure 1A）Or 220（Fig. 2）To execute.Show at some In example, computer-readable recording medium（For example, memorizer 232）Can be with store instruction, module or algorithm, when this instruction, module Or so that one or more processors during algorithm performs（For example, processor 231）Execute the shown step in this flow chart One or more of step.

The method of Figure 14 B includes receiving data packet（1402）, wherein, among other things, this packet can To include data packet header and payload data.For example, payload data can include the user input of such as forwarding The user input data of order, the user input order instruction user input data of this forwarding is to forward from third party device. Source device 120 can include allowing the communication component of transmission packet, and it is included referring for example to transmission unit illustrated in fig. 2 233 and radio modem 234.Then, source device 120 can parse to data packet header（1404）And it is permissible Determine that this payload data includes the user input order forwarding（1404）.Then, source device 120 can be to this packet The payload data including is parsed（1406）, set with the third party corresponding to the user input order forwarding with identifying Standby associated mark.Then, source device 120 can be entered to payload data based on the mark of the third party device being identified Row is processed（1408）.The packet of reference picture 14A and 14B description generally can take the packet with reference to Fig. 6 description Form, and can be used for controlling the audio/video data at source device.

Figure 15 A is according to present disclosure, sends the exemplary method of user data from wireless host device to radio source device Flow chart.Shown exemplary method can be by host device 160（Figure 1A）Or 360（Fig. 3）To execute.In some instances, Computer-readable recording medium（For example, memorizer 332）Can be with store instruction, module or algorithm, when this instruction, module or algorithm So that one or more processors during execution（For example, processor 331）Execute in the shown step in this flow chart Individual or multiple steps.

The method of Figure 15 A includes obtaining user input data at wireless host device（1501）.This user input data can To have associated coordinate data.For example, this associated coordinate data can be with the position of mouse click event or touch The position of event is corresponding.Then, the coordinate data that host device 160 can be associated to this is normalized to generate normalization Coordinate data（1503）.Then, host device 160 can generate including normalized coordinate data packet （1505）.Coordinate data is normalized and can include：Resolution based on display window and the display in source（Such as source sets Standby 120 display 22）The ratio of resolution associated coordinate data is zoomed in and out.The resolution of display window is permissible Determined by host device 160, and can be from the resolution of the display of source device 120 reception source device.Then, host device 160 The packet with normalization coordinate can be sent to radio source device 120（1507）.As the method for Figure 15 A one Point, whether the content that host device 160 can also be for receiving from radio source device determines associated coordinate data in display window Within mouthful, and for example, if associated coordinate data is outside display window, locally processing to user input, Or else if this input is then normalized to coordinate as described within display window.

Figure 15 B is according to present disclosure, showing from wireless host device receiving user's input data at radio source device The flow chart of example method.Shown exemplary method can be by source device 120（Figure 1A）Or 220（Fig. 2）To execute.Show at some In example, computer-readable recording medium（For example, memorizer 232）Can be with store instruction, module or algorithm, when this instruction, module Or so that one or more processors during algorithm performs（For example, processor 231）Execute the shown step in this flow chart One or more of step.

The method of Figure 15 B includes receiving data packet at radio source device, and wherein, this packet includes and number of coordinates According to associated user input data（1502）.For example, this associated coordinate data can with host device at click The position of the position of event or touch event is corresponding.Then, the coordinate data that source device 120 can be associated to this is returned One change, to generate normalized coordinate data（1504）.Source device 120 can be by the resolution based on display window and source The ratio of the resolution of the display coordinate data associated to this zooms in and out, and this coordinate data is normalized.Source Equipment 120 can determine the resolution of the display of source device, and can receive the resolution of display window from wireless host device Rate.Then, source device can be processed to packet based on normalized coordinate data（1506）.Reference picture 15A and The packet of 15B description generally can take the form of the packet with reference to Fig. 6 description, and can be used in source device Place controls audio/video data.

For the sake of simplicity of explanation, reference picture 7-15 is individually described to the various aspects of present disclosure.So And it should envision these various aspects can be combined with each other and mutually combined using and be not only single use.Generally, permissible Realize function described herein and/or module in both radio source device and wireless host device.By this way, current User interface function described in example can between radio source device and wireless host device used interchangeably.

Can use and include wireless handheld device and integrated circuit（IC）Or one group of IC（That is, chipset）Various equipment To realize the technology of present disclosure with device.There is provided described any assembly, module or unit to emphasize feature aspect, And be not necessarily to realize by different hardware cells.

Correspondingly, it is possible to use hardware, software, firmware or their combination in any are realizing technology described herein. If realized with hardware, any feature being described as module, unit or assembly can be realized together in integration logic device In part or individually it is embodied as logical device that is discrete but can co-operating.If realized with software, these technology are permissible At least partly to be realized by computer-readable medium, this computer-readable medium includes executing when executing within a processor above-mentioned The instruction of one or more of method method.Computer-readable medium can include tangible and non-transitory computer can Reading storage medium, and the part of computer program can be formed, computer program can include encapsulating material.Meter Calculation machine readable storage medium storing program for executing can include such as Synchronous Dynamic Random Access Memory（SDRAM）Random access memory （RAM）, read only memory（ROM）, nonvolatile RAM（NVRAM）, Electrically Erasable Read Only Memory （EEPROM）, flash memory, magnetical or optical data storage medium etc..Additionally or alternatively, these technology can at least partly by Realizing, this computer-readable communication media carries the communication with instruction or data structure form to computer-readable communication media Code and can by computer access, read and/or execute.

Code can be by such as one or more digital signal processors（DSP）, general purpose microprocessor, special IC （ASIC）, field programmable logic array（FPGA）Or one or more process of other equivalent integrated or discrete logic circuitry Device is executing.Correspondingly, as used herein, term " processor " also refers to aforementioned structure or is suitable for reality Any one of any other structure of existing technology described herein.Additionally, in certain aspects, can be configured to The dedicated software modules or the interior offer of hardware module that are encoded and decoded or include composite video codec are described herein Function.In addition it is possible to use one or more circuit or logical block comprehensively to realize these technology.

The various aspects of present disclosure are described.These aspects and other side are in following claim Within the scope of.

Claims (20)

1. a kind of method sending user data to radio source device from wireless host device, methods described includes：

Receive the instruction of the resolution of the display to described radio source device from described radio source device；

Obtain user input data at described wireless host device, wherein, described user input data has associated coordinate Data；

At described wireless host device, described associated coordinate data is normalized, to generate normalized number of coordinates According to；

Generate packet, described packet includes data packet header and payload data, wherein, described packet Header includes including the value of general information element for input classification field for indicating described payload data, wherein, Described payload data includes one or more general information elements, and one or more of general information elements include described Normalized coordinate data, and wherein, described value is selection from the list of the input classification supported, described support defeated The list entering classification includes the value for the described value of universal input with for indicating man-machine interface device commands；And

Send described packet from described wireless host device to described radio source device；

Wherein, described coordinate data is normalized including：Aobvious based on the content for receiving from described radio source device Show that the resolution of window is come to the described coordinate being associated with the ratio of the resolution of the described display of described radio source device Data zooms in and out.

2. method according to claim 1, also includes：

Determine described associated coordinate data whether the content for receiving from described radio source device display window it Interior.

3. method according to claim 1, also includes：

Determine the resolution of the described display window for the content receiving from described radio source device.

4. method according to claim 1, wherein, described associated coordinate data corresponds to the position of mouse click event Put.

5. method according to claim 1, wherein, described associated coordinate data corresponds to the position of touch event.

6. a kind of wireless host device for sending user data to radio source device, described wireless host device includes：

Memorizer, its store instruction；

One or more processors, it is configured to execute described instruction, wherein, after executing described instruction, one or many Individual processor makes：

Receive the instruction of the resolution of the display to described radio source device from described radio source device；

Obtain user input data at described wireless host device, wherein, described user input data has associated coordinate Data；

Described associated coordinate data is normalized, to generate normalized coordinate data；

And

Generate packet, described packet includes data packet header and payload data, wherein, described packet Header includes including the value of general information element for input classification field for indicating described payload data, wherein, Described payload data includes one or more general information elements, and one or more of general information elements include described Normalized coordinate data, and wherein, described value is selection from the list of the input classification supported, described support defeated The list entering classification includes the value for the described value of universal input with for indicating man-machine interface device commands；

Wherein, described coordinate data is normalized including：Aobvious based on the content for receiving from described radio source device Show that the resolution of window is come to the described coordinate being associated with the ratio of the resolution of the described display of described radio source device Data zooms in and out；

Transmission unit, it is used for sending described packet to described radio source device.

7. wireless host device according to claim 6, wherein, after executing described instruction, one or more of process Device also makes：

Determine described associated coordinate data whether the content for receiving from described radio source device display window it Interior.

8. wireless host device according to claim 6, wherein, after executing described instruction, one or more of process Device also makes：

Determine the resolution of the described display window for the content receiving from described radio source device.

9. wireless host device according to claim 6, wherein, described associated coordinate data corresponds to click thing The position of part.

10. wireless host device according to claim 6, wherein, described associated coordinate data corresponds to touch event Position.

A kind of 11. wireless host devices for sending user data to radio source device, described wireless host device includes：

For receiving the module of the instruction of the resolution of the display to described radio source device from described radio source device；

For obtaining the module of user input data at described wireless host device, wherein, described user input data has phase The coordinate data of association；

For being normalized to described associated coordinate data, to generate the module of normalized coordinate data；

For generating the module of packet, described packet includes data packet header and payload data, wherein, institute State data packet header to include including general information element for input classification field for indicating described payload data Value, wherein, described payload data includes one or more general information elements, one or more of general informations units Element includes described normalized coordinate data, and wherein, described value is to select from the list of the input classification supported, institute The list stating the input classification of support includes the value for the described value of universal input with for indicating man-machine interface device commands； And

For sending the module of described packet to described radio source device；

Wherein, the described module for being normalized to described coordinate data includes：For based on for from described radio sources The resolution of the display window of content that equipment receives is next with the ratio of the resolution of described display of described radio source device The module that described associated coordinate data is zoomed in and out.

12. a kind of at radio source device from the method for wireless host device receive user data, methods described includes：

Receive the resolution of the display window for the content receiving from described radio source device from described wireless host device；

Receiving data packet at described radio source device, wherein, described packet includes data packet header and effective load Lotus data, wherein, described data packet header is included for input classification field for indicating described payload data bag Include the value of general information element, wherein, described payload data includes one or more general information elements, one or Multiple general information elements include the user input data with associated coordinate data, and wherein, described value be from Select in the list of input classification held, the list of the input classification of described support include described value for universal input with For indicating the value of man-machine interface device commands；

Described associated coordinate data is normalized, to generate normalized coordinate data；And

Based on described normalized coordinate data, described packet is processed；

Wherein, described coordinate data is normalized including：Resolution based on described display window is set with described radio sources The ratio of the resolution of standby display described associated coordinate data is zoomed in and out.

13. methods according to claim 12, also include：

Receive the positional information of described display window from described wireless host device；

Determine the resolution of the display of described source device.

14. methods according to claim 12, wherein, described associated coordinate data corresponds to mouse click event Position.

15. methods according to claim 12, wherein, described associated coordinate data corresponds to the position of touch event Put.

16. a kind of for the radio source device from wireless host device receive user data, described radio source device includes：

Transmission unit, it is used for receiving the display window for the content receiving from described radio source device from described wireless host device The resolution of mouth, and for the packet of receiving data at described radio source device, wherein, described packet includes data and divides Group header and payload data, wherein, described data packet header includes described for indicating for input classification field Payload data includes the value of general information element, and wherein, described payload data includes one or more general informations Element, one or more of general information elements include the user input data with associated coordinate data, and its In, described value is to select from the list of the input classification supported, the list of the input classification of described support is included for logical Value with the described value of input with for indicating man-machine interface device commands；

Memorizer, its store instruction；

One or more processors, it is configured to execute described instruction, wherein, after executing described instruction, one or many Individual processor makes：

Described associated coordinate data is normalized, to generate normalized coordinate data；

And

Based on described normalized coordinate data, described packet is processed；

Wherein, described coordinate data is normalized including：Resolution based on described display window is set with described radio sources The ratio of the resolution of standby display described associated coordinate data is zoomed in and out.

17. radio source devices according to claim 16, wherein, after executing described instruction, one or more of places Reason device also makes：

Receive the positional information of described display window from described wireless host device；

Determine the resolution of the display of described source device.

18. radio source devices according to claim 16, wherein, described associated coordinate data corresponds to click The position of event.

19. radio source devices according to claim 16, wherein, described associated coordinate data corresponds to touch event Position.

20. a kind of for the radio source device from wireless host device receive user data, described radio source device includes：

For receiving the resolution of the display window for the content receiving from described radio source device from described wireless host device Module；

For the module of the packet of receiving data at described radio source device, wherein, described packet includes packet report Head and payload data, wherein, described data packet header includes described effective for indicating for input classification field Load data includes the value of general information element, and wherein, described payload data includes one or more general information elements, One or more of general information elements include the user input data with associated coordinate data, and wherein, institute Stating value is to select from the list of the input classification supported, the list of the input classification of described support is included for universal input Described value and for indicating the value of man-machine interface device commands；

For being normalized to described associated coordinate data, to generate the module of normalized coordinate data；

For the module described packet being processed based on described normalized coordinate data；

Wherein, the described module for being normalized to described coordinate data includes：For dividing based on described display window Resolution and the ratio of the resolution of the display of described radio source device the described coordinate data being associated is zoomed in and out Module.