WO2017128913A1

WO2017128913A1 - Mobile terminal projection method, terminal apparatus and system

Info

Publication number: WO2017128913A1
Application number: PCT/CN2016/112410
Authority: WO
Inventors: 陶冬; 何利鹏; 王佰祥
Original assignee: 努比亚技术有限公司
Priority date: 2016-01-27
Filing date: 2016-12-27
Publication date: 2017-08-03
Also published as: CN105739937B; CN105739937A

Abstract

A mobile terminal projection method, a terminal apparatus and a system, wherein said method is applicable to a mobile terminal, and comprises the following steps: said mobile terminal establishing a communication connection with an external device (S11); the mobile terminal adjusting split-screens, and determining a screen projection area (S12); transmitting screen data with the external device according to the determined screen projection area. On the basis of a Wi-Fi display and split-screen function of a mobile apparatus, a self-defined projection area can be quickly configured for the projection-side device, thereby quickly selecting the desktop area that requires sharing with other devices.

Description

Mobile terminal projection method, terminal device and system

Technical field

The present application relates to, but is not limited to, the field of communications technologies, and in particular, to a mobile terminal projection method, a terminal device, and a system.

Background technique

With the rapid development of the electronics industry, the degree of intelligence of mobile terminals is getting higher and higher. Mobile terminal R&D companies are also paying more and more attention to intelligent and user-friendly design. In the context of the rapid development of mobile terminals, the convenient operation and user-friendly design of the terminal have become a part of the mobile terminal.

At present, mobile terminals are rapidly spreading, and more and more users are beginning to hold two or more mobile devices. Therefore, Wi-Fi Display (WFD) function is born with the wave of technological development, that is, in mobile devices. WFD function. This function can wirelessly project a projection device interface and projection device status to another receiving device, which can then operate the projection device.

However, the current Wi-Fi Display function and the split screen function of the mobile terminal are not effectively combined, which affects the user's real experience.

Summary of the invention

The following is an overview of the topics detailed in this document. This Summary is not intended to limit the scope of the claims.

An embodiment of the present invention provides a mobile terminal projection method, a terminal device, and a system, for delivering a media data stream to an external device, where the external device includes a display screen or a screen, and the method includes:

The mobile terminal establishes a communication connection with an external device;

Determining a projection area of the screen of the mobile terminal;

A media data stream of the projection area is generated and transmitted to the external device.

In an exemplary embodiment, the mobile terminal establishes a communication connection with an external device, Includes the following steps:

The mobile terminal sends a WFD connection request;

Receiving a discovery response at the mobile terminal, wherein the discovery response is sent by an external device in response to a received discovery request;

The mobile terminal establishes a WFD connection with an external device.

In an exemplary embodiment, the determining a projection area of the screen of the mobile terminal includes:

After the WFD connection is successful, the mobile terminal displays a projection area selection frame while projecting, and determines the projection area according to the position of the projection area selection frame.

In an exemplary embodiment, the mobile terminal projection method further includes:

The mobile terminal receives an input signal, wherein the input signal is used to control the display area selection frame to be displayed or closed, and the content in the projection area selection frame is projected when the projection area selection frame is displayed, when the projection area selection frame is closed. Full screen projection.

In an exemplary embodiment, when the mobile terminal supports the upper and lower split screens, the projection area selection frame is displayed with the height as a reference value; when the mobile terminal supports the left and right split screens, the projection area selection frame is referenced by the width. Display; when the mobile terminal supports free split screen, the projection area selection box can be displayed anywhere on the screen.

In an exemplary embodiment, the generating a media data stream of the projection area includes:

The mobile terminal captures image information of the projection area in a preset period, and packs each frame image in time sequence to form an audio/video stream.

In an exemplary embodiment, the generating a media data stream of the projection area and transmitting the data stream to the external device includes:

Sending, by the mobile terminal, a coordinate point of the projection area to the external device;

Converting, by the external device, the abscissa of the determined projection area to a ratio of the abscissa to the first preset value, and converting the ordinate to a ratio of the ordinate to the second preset value; wherein the first preset value For the width of the projection area of the mobile terminal, the second preset value is the height of the projection area of the mobile terminal.

The embodiment of the invention further provides a terminal device, including:

a connection module configured to establish a communication connection between the mobile terminal and an external device;

Determining a module, configured to determine a projection area of the screen of the mobile terminal;

And a transmission module configured to generate a media data stream of the projection area and transmit the same to the external device.

In an exemplary embodiment, the connection module is further configured to send a WFD connection request;

Establish a WFD connection between the mobile terminal and the external device.

In an exemplary embodiment, the determining module is further configured to: after the WFD connection is successful, display a projection area selection frame while the mobile terminal performs projection, and determine the projection area according to the position of the projection area selection frame.

In an exemplary embodiment, the determining module is further configured to receive an input signal, wherein the input signal is used to control a display area selection frame to be displayed or closed, and the content in the selection frame is selected for the projection area when the projection area selection frame is displayed. Projection, full-screen projection when the projection area selection box is closed.

In an exemplary embodiment, the transmission module is configured to generate a media data stream of the projection area by:

The image information of the projection area is captured by the mobile terminal in a preset period, and each frame image is packed in time order to form an audio/video stream.

In an exemplary embodiment, the transmission module is configured to send a coordinate point of the projection area of the mobile terminal to the external device;

An embodiment of the present invention further provides a projection system, including a mobile terminal and one or more external devices, wherein the mobile terminal is configured to establish a communication connection with an external device, and determine a projection area of the screen of the mobile terminal. Generating a media data stream of the projection area and transmitting the same to the external device;

The one or more external devices are configured to transmit a discovery response in response to the received discovery request, and display the media data stream after establishing a connection with the mobile terminal.

In an exemplary embodiment, the external device is further configured to convert the abscissa of the projection area of the mobile terminal to a ratio of the abscissa to the first preset value, and convert the ordinate to the ordinate and the second preset value. a ratio; wherein the first preset value is a width of a projection area of the mobile terminal, and the second preset value is a height of a projection area of the mobile terminal.

In an exemplary embodiment, the mobile terminal is further configured to receive an input signal, wherein the input signal is for controlling display or selection of the projection area selection frame, and the projection is performed when the projection area selection frame is displayed Content projection in the area selection box, and full-screen projection when the projection area selection box is closed;

Wherein, when the mobile terminal supports the upper and lower split screens, the projection area selection frame is displayed with the height as a reference value;

When the mobile terminal supports left and right split screens, the projection area selection frame is displayed with a width as a reference value;

When the mobile terminal supports free split screen, the projection area selection box may be any position of the screen for display.

An embodiment of the present invention further provides a mobile terminal, including: an input device, a display screen, a processor, and a wireless communication unit;

The wireless communication unit is configured to establish a communication connection between the mobile terminal and an external device;

The input device is configured to receive an input signal and transmit to the processor;

The processor is configured to determine a projection area of the display screen based on the input signal, generate a media data stream of the projection area, and transmit the media data stream to the external device.

In an exemplary embodiment, the mobile terminal further includes: an audio unit configured to: The media data stream of the projection area generated by the processor is converted into an audio signal and output.

In an exemplary embodiment, the input device is a touch screen.

A mobile terminal projection method according to an embodiment of the present invention can realize one-to-one or one-to-many screen projection and sharing, and can reduce dependence on a place through a WFD network, and can be applied to a multimedia conference room or an audio-visual function classroom. In the scenario of demand, the WFD network can be quickly formed to realize resource sharing. Based on the Wi-Fi Display and split screen function in the mobile device, the projection side device can be quickly set to customize the projection area to quickly select the desktop area to be shared with other devices. The implementation of the method overcomes the current Wi-Fi Display function to project the entire projection desktop to the receiving end, which affects the user experience. At the same time, it can well reflect the humanized design concept of mobile terminals.

Other aspects will be apparent upon reading and understanding the drawings and detailed description.

BRIEF abstract

The present application will be further described below in conjunction with the accompanying drawings and embodiments, in which:

1 is a schematic structural diagram of hardware of an exemplary mobile terminal implementing an embodiment of the present application;

2 is a search flow chart of a projection side device according to an embodiment of the present invention;

3 is a flowchart of a method for projecting a mobile terminal according to an embodiment of the present invention;

4 is a schematic diagram of a terminal projected by a mobile terminal according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a network structure capable of implementing multi-screen projection control according to an embodiment of the present invention; FIG.

6 is a schematic diagram of a projection side device turning on multiple processes to perform screen data transmission control according to an embodiment of the present invention;

Figure 7 is a schematic diagram of the control effect shown in Figure 6;

8 is a structural block diagram of a terminal device according to an embodiment of the present invention;

9 is a schematic diagram of an interaction interface of a terminal device according to an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of a projection system of a mobile terminal according to an embodiment of the present invention; FIG.

FIG. 11 is a schematic structural diagram of a mobile terminal that implements projection in an embodiment of the present invention.

Embodiments of the invention

It is understood that the embodiments described herein are merely illustrative of the application and are not intended to limit the application.

A mobile terminal implementing an embodiment of the present application will now be described with reference to the accompanying drawings. In the following description, the use of suffixes such as "module", "component" or "unit" for indicating an element is merely an explanation for facilitating the present application, and does not have a specific meaning per se. Therefore, "module" and "component" can be used in combination.

The mobile terminal can be implemented in various forms. For example, the terminals described in this application may include, for example, mobile phones, smart phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablets), PMPs (Portable Multimedia Players), navigation devices, and the like. Mobile terminals and fixed terminals such as digital TVs, desktop computers, and the like. In the following, it is assumed that the terminal is a mobile terminal. However, those skilled in the art will appreciate that configurations in accordance with embodiments of the present application can be applied to fixed type terminals in addition to elements that are specifically for mobile purposes.

Example 1

FIG. 1 is a schematic structural diagram of hardware of a terminal device according to an embodiment of the present invention.

The terminal device 100 of the embodiment of the present invention includes: a probe connection unit 101, a control unit 102, and a display unit 103.

When the terminal device 100 is acting as a projection side (Source side):

The probe connection unit 101 is configured to perform device discovery to search for neighboring other devices available to the Wi-Fi Display service. When the other devices are found, the probe connection unit 101 performs interaction and capability negotiation with other devices through the Rapid Spanning Tree Protocol (RTSP) protocol to connect the terminal device 100 with other devices.

The control unit 102 controls basic operations within the terminal device 100. The control unit 102 performs control so that the content (screen data) displayed by the display unit 103 is transmitted to other devices through the probe connection unit 101 based on RTP (Real-time Transport Protocol). RTP is the real-time transport protocol. The control unit 102 determines the input signal form from the user through the display unit 103 to control the terminal device 100 in accordance with the input signal. For example, a device search or the like is initiated based on an input signal.

When the terminal device 100 is acting as the receiving side (Sink side):

The probe connection unit 101 is further configured to establish a connection with the device on the projection side to receive screen data. The control unit 102 is also used to perform control such that the display unit 103 displays the received screen data.

2 is a flowchart of searching for a projection side device according to an embodiment of the present invention. The search process of the embodiment of the present invention includes:

Device Discovery Step: The projection side device performs device discovery for finding neighboring external devices that support Wi-Fi Display.

Device connection step: When any other device is found, the projection side device sets a connection with the device, performs switching capability, and performs negotiation.

Playback step: the device uses the RTSP protocol to perform capability negotiation and match related parameters. After the RTSP play command, the current screen data of the projection side device is first packaged into an audio and video stream, and then the audio stream packet is directly sent to the receiving side through the RTP protocol. The device, the screen data on the projection side is transmitted to the receiving side by stream type (audio/video (AV) stream). RTSP is a real-time transport protocol, which belongs to an application layer protocol in the TCP/IP protocol system and is mainly responsible for device capability negotiation.

Example 2

Based on the hardware structure of the terminal device and the search flow of the projection side terminal, a projection method of the device split screen of the embodiment of the present invention is proposed. As shown in FIG. 3, in the embodiment of the present invention, a projection method of a device split screen is applied to a mobile terminal, and includes the following steps:

In step S11, the mobile terminal establishes a communication connection with an external device.

It should be understood that the mobile terminal is equivalent to a projection side device such as a mobile phone, a PAD (tablet computer) or the like, and the external device is equivalent to a receiving side device, and the external device includes a display screen or a screen such as a mobile phone, a television, a projector, and the like. Under certain conditions, the mobile terminal can be converted into an external device, that is, converted into a receiving device, and the external device can be converted into a mobile terminal, that is, converted into a projection device. In this embodiment, the projection device supports the split screen function, and The receiving side device is not limited. The split screen function refers to dividing the screen of the mobile terminal into several areas, and each area is a separate display area, and can also implement a split screen in the form of a single window, a double window, a floating window, etc., for example, in a split screen state. The next user can open a Taobao shopping while chatting with friends on WeChat.

The projection side device and the receiving side device can be directly connected to each other through Wi-Fi direct without using a wireless local area network (WLAN) access point (AP) or the like. The projection side device finds the receiving side device. During device discovery, the projection side device sends a probe request frame on a predefined channel. The probe request frame includes information such as a peer-to-peer (P2P) information element (IE), a P2P wildcard SSID, a wildcard SSID, a destination address, a device type, and a device ID. The receiving side device listens to the probe request frame on a predefined channel. Each device determines that the information in the probe request frame matches the search criteria. If a match is found, the receiving side device transmits a probe request frame having the device description to the projection side device.

Step S12, determining a projection area of the screen of the mobile terminal.

After the projection side device enters the normal connection projection through the WFD, the projection side device connects the entire screen to the receiving side device by default, and at the same time, the projection side device displays a projection area selection frame of the selectable projection, and the position of the projection area selection frame can be The desktop condition of the projection side device is adjusted. The projection area selection box is equal to or smaller than the current desktop application icon size by default, and the user can manually adjust the size of the projection area selection box. According to the split screen design of the projection side device, the display area selection frame may include: the projection side device only supports the upper and lower split screens, and the projection area selection frame is displayed with the height as a reference value; the projection side device only supports the left and right points. In the screen, the projection area selection box is displayed with the width as a reference value; the projection side device supports freely selecting the projection area, and the projection area selection box can be displayed at any position of the screen. As shown in FIG. 4, the projection device of the embodiment of the present invention can customize a projection area, and the rectangular dotted frame of the device 200 in the figure represents a projection area selection frame that can be dragged to change at least one of the position and the size, and the projection area selection box. The inside P indicates a projection area, and the projection side device transmits the content of the projection area P to the receiving side device, and the unprojected area Q is not affected by the projection area P.

In this embodiment, the projection side device is configured to trigger the projection area selection box to display or hide the switch. For example, when displaying, the projection area selection frame can be set to press the time t, and T<t, the projection area selection frame is closed, and the projection is closed. The side device performs full screen projection. When the projection is performed, the projection side device can be set to press and hold a certain area time t, and T < t, the projection area selection frame is displayed. Similarly, according to the situation of the projection side device, a touch button or a button for triggering the projection area selection box is set.

In step S13, a media data stream of the projection area is generated and transmitted to the external device.

The projection side device utilizes the underlying wireless chain according to the connection created when the connection is established. The road packs the current screen video and audio data and transmits it to the corresponding receiving side device in combination with the RTP protocol. The projection side device captures the current screen data every preset time (for example, 30 ms), packages it into an audio and video stream, and transmits the screen data to the receiving side device according to the RTP protocol. The receiving side device is mainly responsible for displaying the screen data transmitted by the projection side device. The receiving side device caches and displays the received media data stream containing the screen data. In another embodiment, the projection side device screens the screen and crops each frame according to the size of the screen projection area, only retains the image of the screen projection area, and then makes the cropped screenshot into an audio/video stream. , sent to an external device.

In order to enable the projection area in the projection side device to adaptively display the projection size of the receiving device screen size, the receiving side device converts the abscissa of the projection area of the projection side device into a ratio of the abscissa to the first preset value, and the projection area The ordinate is converted into a ratio of the ordinate to the second preset value; wherein the first preset value is a width of a projection area of the mobile terminal, and the second preset value is a height of a projection area of the mobile terminal. For example, if the projection area of the projection side device has a width of W0 and a height of H0, and a coordinate point is (X0, Y0), it is converted into (X0/W0, Y0/H0), that is, it is converted into occupied. The ratio of the projection area of the mobile terminal in the width direction and the length direction. In addition, conversion can also be performed in other ways. If the width of the screen of the receiving device is W1, the height is H1, and the received coordinate is (X0/W0, Y0/H0), the coordinates of the point of the receiving device correspond to the coordinates of the projection device ((X0) /W0) × W1, (Y0/H0) × H1). As shown in FIG. 4, after the device 200 determines that P is a projection area, the content of the projection area P is projected to the device 300, and the device 300 adjusts according to the size of the projection area and the display area of the device itself. In this embodiment, the receiving side device 300 displays the received projection content in full screen, and cannot view other unprojected areas of the projection side device. It should be understood that the receiving side device 300 can also be adjusted to display the received projected content in a certain area.

It should be understood that the projection method given in this embodiment is not limited to the above sequence of steps, and the mobile terminal may first determine the projection area of the screen, and then establish a communication connection with the external device.

The mobile terminal projection method of the embodiment of the invention can realize one-to-one screen projection and sharing, and can reduce the dependence on the location through the WFD network, and can be applied to a specific demand scene such as a multimedia conference room or an audio-visual function classroom. To quickly build a WFD network and achieve resource sharing. Based on the Wi-Fi Display and split screen function in the mobile device, the projection side device can be quickly set to customize the projection area to quickly select the desktop area to be shared with other devices. The implementation of the method, Overcoming the current Wi-Fi Display function, the entire projection desktop is projected to the receiving end, which affects the user experience. Moreover, the receiving side device only operates or views the projection area of the projection side device, and cannot view other unprojected areas of the projection side device, and the unprojected area of the projection side device is not affected by the projection area.

Example 3

Based on the above embodiments, the embodiment of the present invention further provides a multi-screen projection method, which has one or more receiving side devices with respect to one projection side device. FIG. 5 is a schematic structural diagram of a network capable of implementing multi-screen projection control according to an embodiment of the present invention.

The network 1000 includes: one projection side device 200 and at least one reception side device 300. The hardware configuration of the projection side device 200 and the reception side device 300 can be referred to the above FIG.

Network 1000 has the same meaning as a Wi-Fi peer-to-peer (P2P) network. The projection side device 200 and the reception side device 300 in the network 1000 are directly connected to each other through Wi-Fi direct without using a wireless local area network (WLAN) access point (AP) or the like.

The projection side device 200 performs device discovery in the network 1000 environment to discover the reception side device 300. During device discovery, the projection side device 200 transmits a probe request frame on a predefined channel. The probe request frame includes information such as a peer-to-peer (P2P) information element (IE), a P2P wildcard SSID, a wildcard SSID, a destination address, a device type, and a device ID. The receiving side device 300 in the network 1000 environment listens to the probe request frame on a predefined channel. Each device determines that the information in the probe request frame matches the search criteria. If a match is found, the receiving side device 300 transmits a probe request frame having the device description to the projection side device 200.

After the device discovery is completed, the projection side device 200 performs service discovery to obtain a service supported by the reception side device 300. During the service discovery, the projection side device 200 transmits a GAS initial request to the receiving side device 300. In response, the receiving side device 300 transmits the GAS initial response with the service information to the projection side device 200. Thereafter, the projection side device 200 establishes a WFD connection with the reception side device 300 using the device and service description obtained from the reception side device 300, that is, the device connection establishment in FIG.

It should be understood that each receiving side device 300 in the embodiment of the present invention establishes a connection with the projection side device 200 in the above manner. Thereby, the projection side device 200 and the plurality of receiving side devices 300 can be Called a group. The projection side device 300 is designated as a group owner (GO). The receiving side device 300 functions as a group client.

Referring to FIG. 6, device A is a projection side device, and device B, device C, and device D are both receiving side devices. Device B, device C, and device D are connected to device A through Wireless.

Device A starts multiple process services, and each process manages communication with one receiving device separately. For example, the sub-process 1 corresponds to the communication connection data of the processing device B, the sub-process 2 corresponds to the communication connection data of the processing device C, and the sub-process 3 corresponds to the communication connection data of the processing device D. In addition, device A also starts a parent process for listening connections and so on.

In the embodiment of the present invention, each process is transmitted according to the WFD protocol.

In order to implement the underlying communication link, the device A allocates an IP address to each receiving device, and stores the correspondence between each receiving device and its IP address in an IP list.

In the embodiment of the present invention, any device connected to the device A through the IP list can access other devices through the IP list, establish a TCP/IP connection, and implement two-way communication, so that any device can display screen data. Passed to other devices over a TCP/IP link. Thereby, screen mutual investment between devices can be achieved.

The screen data transfer process between the entire group includes:

(1) Connection establishment

On the underlying link, the projection side device performs the discovery process of the receiving side device according to the above-mentioned execution, and listens for the connection request from other devices; and selects the specified device to establish a connection according to the discovered device. When a connection is established with a plurality of (two or more) devices, the projection side device creates a plurality of sub-processes to process the bidirectional data transmission with each device separately.

(2) Screen data transmission

After the connection is established, the projection side device uses the underlying Wireless link to package the current screen video and audio data according to the process service created when the connection is established, and transmits it to all receiving side devices in combination with the RTP protocol. The projection side device captures the current screen data every preset time (for example, 30 ms), packs it into an audio and video stream, and sends the screen data to the receiving side device according to the RTP protocol. In another embodiment, the projection side device interval screens the screen and according to the screen projection area The size of each frame is cropped, leaving only the image of the projected area of the screen, and then the cropped screenshot is made into an audio/video stream and sent to an external device.

The receiving side device is mainly responsible for displaying the screen data transmitted by the projection side device. The receiving side device caches and displays the received media data stream containing the screen data. Referring to Figure 7, after the screen data begins to be transmitted, devices B, C, and D receive the screen data of device A and display it.

In the embodiment of the present invention, the projection side device manages the list of devices that establish the connection to ensure information security, and prevent irrelevant devices from joining, and also manages the joining of new devices in the middle or the exit of devices in the link.

The multi-screen projection control method of the embodiment of the invention can realize one-to-many screen projection and sharing, and can reduce the dependence on the location through the WFD network; and can be applied to a specific demand scene such as a multimedia conference room or an audio-visual function classroom. Quickly set up a WFD network to achieve resource sharing.

Example 4

FIG. 8 is a structural block diagram of a terminal device according to an embodiment of the present invention, including: a connection module 11, a determining module 12, and a transmission module 13, wherein

The connection module 11 is arranged to establish a communication connection between the mobile terminal and an external device.

It should be understood that the mobile terminal is equivalent to the projection side device, and the external device is equivalent to the receiving side device. Under certain conditions, the mobile terminal can be converted into an external device, that is, converted into a receiving device, and the external device can be converted into a mobile terminal, that is, converted into a projection device. In this embodiment, the projection device supports the split screen function, and The receiving side device is not limited.

The determining module 12 is configured to determine a projected area of the screen of the mobile terminal.

After the projection side device enters the normal connection projection through the WFD, the projection side device connects the entire screen to the receiving side device by default, and at the same time, the projection side device displays a projection area selection frame of the selectable projection, and the position of the projection area selection frame can be The desktop condition of the projection side device is adjusted. The projection area selection box is equal to or smaller than the current desktop application icon size by default, and the user can manually adjust the size of the projection area selection box. According to the split screen design of the projection side device, the display area selection frame may include: the projection side device only supports the upper and lower split screens, and the projection area selection frame is displayed with the height as a reference value; the projection side device only supports the left and right points. In the screen, the projection area selection box is displayed with the width as a reference value; the projection side device supports freely selecting the projection area, and the projection area selection box can be displayed at any position of the screen.

The transmission module 13 is configured to generate a media data stream of the projection area and transmit it to an external device.

The projection side device uses the underlying Wireless link to package the current screen video and audio data according to the connection created when the connection is established, and transmits it to the corresponding receiving side device in combination with the RTP protocol. The projection side device captures the current screen data every preset time (for example, 30 ms), packages it into an audio and video stream, and transmits the screen data to the receiving side device according to the RTP protocol. The receiving side device is mainly responsible for displaying the screen data transmitted by the projection side device. The receiving side device caches the received media data stream containing the screen data. In another embodiment, the projection side device screens the screen and crops each frame according to the size of the screen projection area, only retains the image of the screen projection area, and then makes the cropped screenshot into an audio/video stream. , sent to an external device.

In order to enable the projection area in the projection side device to adaptively display the projection size of the receiving device screen size, the receiving side device converts the abscissa of the projection area of the projection side device into a ratio of the abscissa to the first preset value, and the projection area The ordinate is converted into a ratio of the ordinate to the second preset value; wherein the first preset value is a width of a projection area of the mobile terminal, and the second preset value is a height of a projection area of the mobile terminal. For example, the projection area of the projection side device has a width of W0 and a height of H0. When a coordinate point is (X0, Y0), it is converted into (X0/W0, Y0/H0), that is, it is converted into a ratio of the projection area of the occupied mobile terminal in the width direction and the length direction. In addition, conversion can also be performed in other ways. If the width of the screen of the receiving device is W1, the height is H1, and the received coordinate is (X0/W0, Y0/H0), the coordinates of the point of the receiving device correspond to the coordinates of the projection device ((X0) /W0) × W1, (Y0/H0) × H1).

It should be understood that the implementation principles and details described in the above embodiments 1 to 3 are applicable to the embodiment.

A terminal device of the embodiment of the invention can realize one-to-one screen projection and sharing, and can reduce the dependence on the location through the WFD network, and can be applied to a specific demand scene such as a multimedia conference room or an audio-visual function classroom. Quickly set up a WFD network to achieve resource sharing. Based on the Wi-Fi Display and split screen function in the mobile device, the projection side device can be quickly set to customize the projection area to quickly select the desktop area to be shared with other devices. The implementation of the method overcomes the current Wi-Fi Display function to project the entire projection desktop to the receiving end, which affects the user experience. At the same time, it can well reflect the humanized design concept of mobile terminals.

Example 5

The embodiment of the invention further provides a terminal device having a hardware structure as shown in FIG. 1 . The terminal device includes: a connection module configured to establish a communication connection between the mobile terminal and the external device; a determining module configured to adjust a split screen of the mobile terminal to determine a projection area of the screen; and a transmission module configured to The determined projection area is used for screen data transmission and display with external devices.

Referring to FIG. 9, in the embodiment of the present invention, the terminal device can be used as a projection side device or as a receiving side device. When the screen projection function is turned on, the user can see the interactive interface as shown on the left side of FIG. 9 through the display unit. When the user selects a "projection screen" through the projection area of the mobile terminal, the interaction interface jumps to the interface on the upper right side, and the terminal device acts as a projection side device, performs a device discovery step, and searches for an adjacent external other supporting Wi-Fi Display. device. When other devices are discovered, the device list is displayed, and a connection is established with the selected user device according to the user's selection, and the exchange capability is performed using RTSP and negotiation is performed. After the connection is established, the terminal device directly transmits the screen data of the projection side device to the receiving side device after the RTSP play command; when receiving the response from the receiving side, starts the audio/video (AV) stream for mirroring, and the projection is performed. The screen data of the side is transmitted to the receiving side device by the stream type.

When the user selects "receive picture" through the projection area of the mobile terminal, the terminal device acts as a receiving side device, waits for the projection party to initiate an invitation, and when the invitation is received, that is, after establishing a connection with the projection side, the screen data sent by the projection side can be received.

A terminal device of the embodiment of the invention can realize one-to-one screen projection and sharing, and can reduce the dependence on the location through the WFD network, and can be applied to a specific demand scene such as a multimedia conference room or an audio-visual function classroom. Quickly set up a WFD network to achieve resource sharing. Based on the Wi-Fi Display and split screen function in the mobile device, the projection side device can be quickly set to customize the projection area, so that the desktop area to be shared with other devices can be quickly selected, and multi-screen interaction can be realized, and the role can be switched. A device can be shared to enhance the user experience.

Example 6

Referring to FIG. 10, an embodiment of the present invention further provides a projection system. The system includes:

The mobile terminal 701 is configured to establish a communication connection with an external device, determine a screen projection area of the mobile terminal, generate an audio/video stream of the screen projection area, and transmit the audio/video stream to the external device;

The one or more external devices 702 are configured to transmit a discovery response in response to the received discovery request, displaying the audio/video stream after establishing a connection with the mobile terminal.

The mobile terminal is equivalent to a projection side device, such as a mobile phone, a PAD (tablet computer), etc., and the external device is equivalent to a receiving side device, and the external device includes a display screen or a screen, such as a mobile phone, a television, a projector, and the like. Under certain conditions, the mobile terminal can be converted into an external device, that is, converted into a receiving device, and the external device can be converted into a mobile terminal, that is, converted into a projection device. In this embodiment, the projection device supports the split screen function, and The receiving side device is not limited.

The mobile terminal 701 is also arranged to send a WFD connection request to establish a WFD connection with one or more external devices.

The mobile terminal 701 is further configured to adjust the split screen, determine a projection area of the screen, and perform screen data transmission with the external device according to the determined projection area. Wherein the projection area of the determining screen comprises:

When the mobile terminal supports the upper and lower split screens, the projection area selection frame is displayed with the height as a reference value;

When the mobile terminal supports the left and right split screens, the projection area selection frame is displayed with the width as a reference value;

When the mobile terminal supports free split screen, the projection area selection box can be displayed anywhere in the screen.

One or more external devices 702 are arranged to receive and display the received screen data.

The external device 702 is further configured to convert the abscissa of the determined projection area into a ratio of the abscissa to the first preset value, and convert the ordinate to a ratio of the ordinate to the second preset value; wherein the first The preset value is the width of the projection area of the mobile terminal, and the second preset value is the height of the projection area of the mobile terminal.

In order to enable the projection area in the projection side device to adaptively display the projection size of the receiving device screen size, the receiving side device converts the abscissa of the projection area of the projection side device into a ratio of the abscissa to the first preset value, and the projection area The ordinate is converted into a ratio of the ordinate to the second preset value; wherein the first preset value is a width of a projection area of the mobile terminal, and the second preset value is a height of a projection area of the mobile terminal. For example, if the projection area of the projection side device has a width W0 and a height of H0 and a coordinate point is (X0, Y0), it is converted into (X0/W0, Y0/H0), that is, it is converted into a occupied movement. The ratio of the projection area of the terminal in the width direction and the length direction. In addition, conversion can also be performed in other ways. If the screen of the receiving device is W1, the height is H1, and the received coordinates are (X0/W0, Y0/H0), the coordinates of the point of the receiving device correspond to the coordinates of the projection device ((X0/). W0) × W1, (Y0/H0) × H1).

The mobile terminal 701 includes a display screen and the display screen includes a first display area and a second display area; the one or more external devices 702 include a display screen, and the projection area determined according to the mobile terminal The field is projected.

In this embodiment, the mobile terminal supports the split screen function, and no restrictions are imposed on the external device. It should be understood that the implementation principles and details described in the above embodiments 1 to 3 are applicable to the embodiment.

The device split screen projection system of the embodiment of the invention can realize one-to-one or one-to-many screen projection and sharing, and can reduce the dependence on the location through the WFD network, and can be applied to a multimedia conference room or an audio-visual function classroom. In a scenario with specific requirements, the WFD network can be quickly formed to achieve resource sharing. Based on the Wi-Fi Display and split screen function in the mobile device, the projection side device can be quickly set to customize the projection area to quickly select the desktop area to be shared with other devices. The implementation of the method overcomes the current Wi-Fi Display function to project the entire projection desktop to the receiving end, which affects the user experience. Moreover, the humanized design concept of the mobile terminal can be well reflected.

Example 7

Referring to FIG. 11, the mobile terminal of the embodiment of the present invention includes: an input device, a processor 903, a display screen 904, a wireless communication unit 905, and an audio unit 906. In one embodiment, the input device is a touch screen 2010. The touch screen 2010 includes a touch panel 901 and a touch controller 902. Further, the input device may also be a non-touch input device (eg, an infrared input device, etc.) or the like.

The wireless communication unit 905 can include a short-range communication module (e.g., a WiFi chip) for the mobile terminal to establish a communication connection with an external device.

Touch controller 902 can be a single application specific integrated circuit (ASIC), which can include one or more processor subsystems, which can include one or more ARM processors or other processors with similar functions and capabilities.

The touch controller 902 is mainly used for receiving a touch signal generated by the touch panel 901, and processing the same to the processor 903 of the mobile terminal. Such processing can determine the projection area of the screen of the mobile terminal, for example, by analog-to-digital conversion of the physical input signal, processing to obtain touch point coordinates, processing to obtain touch duration, and the like.

The processor 903 receives the output of the touch controller 902, performs processing, and performs an action based on the output. The actions include, but are not limited to, capturing image information of a projection area within a preset period (eg, 30 ms), packing each frame image in time sequence to form an audio/video stream, and transmitting to the external device, etc. Wait.

The processor 903 is also coupled to the display screen 904. Display 904 is used to provide a UI to a user of the device.

The audio unit 906 is for converting the screen projection area audio/video stream generated by the processor 903 into an audio signal and outputting it as a sound.

In some embodiments, processor 903 can be a separate component from touch controller 902. In other embodiments, the processor 903 can be a composite component with the touch controller 902.

In one embodiment, the touch panel 901 is provided with a discrete capacitive sensor, a resistive sensor, a force sensor, an optical sensor, or the like.

The touch panel 901 includes an electrode array made of a conductive material in a lateral direction and a longitudinal direction. For a single-touch screen of an M-row and N-column electrode array (only the coordinates of a single touch can be determined), the touch controller 902 uses a self-capacitance scan, and then scans the M rows and the N columns respectively, according to each row and each column. The signal is used to calculate the coordinates of the finger on the touch screen. The number of scans is M+N times.

For a multi-touch touch screen of M rows and N columns of electrode arrays (which can detect and resolve coordinates of multiple points, ie multi-touch), the touch controller 902 uses multi-contact mutual capacitance scanning to intersect the rows and columns. Scanning, whereby the number of scans is M × N times.

When the user's finger touches the panel, the touch panel generates a touch signal (which is an electrical signal) to the touch controller 902. The touch controller 902 can obtain the coordinates of the touched point by scanning. In one embodiment, the touch panel 901 of the touch screen 2010 is physically a set of independent coordinate positioning systems. After the touch point coordinates of the touch are reported to the processor 903, the processor 903 is converted to the display screen 904. Pixel coordinates to correctly identify the input operation.

In the description of the present application, it is to be understood that the terms "first", "second" and the like are used for the purpose of description only and are not to be construed as indicating or implying relative importance. Further, in the description of the present application, the meaning of "a plurality" is two or more unless otherwise stated.

It should be understood that the terminal device, the mobile terminal, and the external device of the embodiments of the present invention may be implemented in various forms. For example, the devices described in this application may include, for example, mobile phones, mobile phones, smart phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablets), PMPs (Portable Multimedia Players), navigation devices Mobile devices such as the like and fixed devices such as digital TVs, desktop computers, and the like.

Those of ordinary skill in the art will appreciate that all or some of the steps, systems, and functional blocks/units of the methods disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical units; for example, one physical component may have multiple functions, or one function or step may be composed of several physical The components work together. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage medium includes volatile and nonvolatile, implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules or other data. Sex, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or may Any other medium used to store the desired information and that can be accessed by the computer. Moreover, it is well known to those skilled in the art that communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. .

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the embodiments described above, and the foregoing embodiments are merely illustrative and not restrictive, and those of ordinary skill in the art In the apocalypse, many forms may be made without departing from the scope of the invention as claimed and claimed.

Industrial applicability

The embodiment of the present invention provides a mobile terminal projection method, a terminal device, and a system, which overcomes the current Wi-Fi Display function, and projects the entire projection end desktop to the receiving end, thereby affecting the user experience, and can well reflect the humanity of the mobile terminal. Design concept.

Claims

A mobile terminal projection method for delivering a media data stream to an external device, the external device comprising a display screen or a screen, the mobile terminal projection method comprising the following steps:

The mobile terminal establishes a communication connection with an external device;

Determining a projection area of the screen of the mobile terminal;

A media data stream of the projection area is generated and transmitted to the external device.
The mobile terminal projection method according to claim 1, wherein the mobile terminal establishes a communication connection with an external device, comprising the steps of:

The mobile terminal sends a wireless network to display a WFD connection request;

Receiving a discovery response at the mobile terminal, wherein the discovery response is sent by an external device in response to a received discovery request;

The mobile terminal establishes a WFD connection with the external device.
The mobile terminal projection method according to claim 2, wherein the determining a projection area of the screen of the mobile terminal comprises:

After the WFD is successfully connected, the mobile terminal displays a projection area selection frame while projecting, and determines a projection area according to the position of the projection area selection frame.
The mobile terminal projection method according to claim 3, wherein the mobile terminal projection method further comprises:

The mobile terminal receives an input signal, wherein the input signal is used to control the display area selection frame to be displayed or closed, and when the projection area selection frame is displayed, the content in the selection area of the projection area is selected for projection. Full-screen projection is performed when the projection area selection box is closed.
The mobile terminal projecting method according to claim 4, wherein

When the mobile terminal supports the upper and lower split screens, the projection area selection frame is displayed with the height as a reference value;

When the mobile terminal supports left and right split screens, the projection area selection frame is displayed with a width as a reference value;

When the mobile terminal supports free split screen, the projection area selection box may be any position of the screen for display.
The mobile terminal projection method according to claim 3, wherein the generating the media data stream of the projection area comprises:

The mobile terminal captures image information of the projection area in a preset period, and packs each frame image in time sequence to form an audio/video stream.
The mobile terminal projection method according to claim 1, wherein the generating a media data stream of the projection area and transmitting the data stream to the external device comprises:

Sending, by the mobile terminal, a coordinate point of the projection area to the external device;

Converting, by the external device, the abscissa of the determined projection area to a ratio of the abscissa to the first preset value, and converting the ordinate of the determined projection area into a ratio of the ordinate to the second preset value; wherein A preset value is a width of a projection area of the mobile terminal, and the second preset value is a height of a projection area of the mobile terminal.
A terminal device comprising:

a connection module configured to establish a communication connection between the mobile terminal and an external device;

Determining a module, configured to determine a projection area of the screen of the mobile terminal;

And a transmission module configured to generate a media data stream of the projection area and transmit the same to the external device.
The terminal device according to claim 8, wherein the connection module is configured to send a WFD connection request;

Receiving a discovery response at the mobile terminal, wherein the discovery response is sent by an external device in response to a received discovery request;

Establishing a WFD connection between the mobile terminal and an external device.
The terminal device according to claim 9, wherein the determining module is configured to: after the WFD connection is successful, display a projection area selection frame while the mobile terminal performs projection, and select a position of the frame according to the projection area, Determine the projection area.
The terminal device according to claim 10, wherein said determining module is further configured to Receiving an input signal, wherein the input signal is used to control display or selection of the projection area selection frame, and when the projection area selection frame is displayed, projecting content in the selection area of the projection area, when the projection area is selected Full screen projection when the box is closed.
The terminal device according to claim 11, wherein

When the mobile terminal supports the upper and lower split screens, the projection area selection frame is displayed with the height as a reference value;

When the mobile terminal supports left and right split screens, the projection area selection frame is displayed with a width as a reference value;

When the mobile terminal supports free split screen, the projection area selection box may be any position of the screen for display.
The terminal device according to claim 8, wherein the transmission module is configured to generate a media data stream of the projection area by:

The mobile terminal captures image information of the projection area in a preset period, and packs each frame image in time sequence to form an audio/video stream.
The terminal device according to claim 8, wherein the transmission module is configured to transmit a coordinate point of the projection area of the mobile terminal to the external device;

Converting, by the external device, the abscissa of the determined projection area to a ratio of the abscissa to the first preset value, and converting the ordinate of the determined projection area into a ratio of the ordinate to the second preset value; wherein A preset value is a width of a projection area of the mobile terminal, and the second preset value is a height of a projection area of the mobile terminal.
A projection system comprising a mobile terminal and one or more external devices, wherein

The mobile terminal is configured to establish a communication connection with an external device, determine a projection area of the screen of the mobile terminal, generate a media data stream of the projection area, and transmit the media data stream to the external device;

The one or more external devices are configured to transmit a discovery response in response to the received discovery request, and display the media data stream after establishing a connection with the mobile terminal.
The projection system of claim 15 wherein said external device is further configured to The mobile terminal determines that the abscissa of the projection area is converted into a ratio of the abscissa to the first preset value, and converts the ordinate to a ratio of the ordinate to the second preset value; wherein the first preset value is a projection of the mobile terminal The width of the area, the second preset value is the height of the projection area of the mobile terminal.
The projection system of claim 15 wherein said mobile terminal is further configured to receive an input signal, wherein said input signal is for controlling said display area selection frame to be displayed or closed when said projection area selection frame is displayed And projecting the content in the frame in the projection area, and performing full-screen projection when the projection area selection box is closed;

Wherein, when the mobile terminal supports the upper and lower split screens, the projection area selection frame is displayed with the height as a reference value;

When the mobile terminal supports left and right split screens, the projection area selection frame is displayed with a width as a reference value;

When the mobile terminal supports free split screen, the projection area selection box may be any position of the screen for display.
A mobile terminal includes: an input device, a processor, a wireless communication unit, and a display screen;

The wireless communication unit is configured to establish a communication connection between the mobile terminal and an external device;

The input device is configured to receive an input signal and transmit to the processor;

The processor is configured to determine a projection area of the display screen based on the input signal, generate a media data stream of the projection area, and transmit the media data stream to the external device.
The mobile terminal of claim 18, further comprising: an audio unit configured to convert the media data stream of the projection area generated by the processor into an audio signal and output.
The mobile terminal of claim 18, wherein the input device is a touch screen.