CN116112748A - Mobile terminal, display equipment and method for adjusting screen throwing picture direction - Google Patents

Abstract

The application provides a mobile terminal, display equipment and a method for adjusting the screen projection direction. And responding to a camera opening event, detecting the direction information of the mobile terminal relative to the vertical plane according to the pose information, and sending the direction information to the display equipment so that the display equipment displays the screen throwing data according to the direction information. The display device can acquire the horizontal and vertical screen states or the placement angles of the mobile terminal by analyzing the direction information, and rotate the picture display control according to the angle information. Thus, the display of the screen-throwing data can improve the viewing experience of the user. In addition, the mobile terminal can also perform angle rotation and other processing on the screen projection data at the data source end, the display equipment can display the screen projection data after receiving the screen projection data sent by the mobile terminal and decoding the screen projection data, and the viewing experience of a user is improved while the efficiency of displaying the screen projection data is improved.

Description

Mobile terminal, display equipment and method for adjusting screen throwing picture direction

Technical Field

The application relates to the technical field of display equipment, in particular to a mobile terminal, display equipment and a method for adjusting the screen projection direction.

Background

The mirror image screen projection refers to that the mobile terminal is connected to the display device in a wired or wireless mode, so that a screen displayed in the mobile terminal is projected into the display device, and the display device displays content corresponding to screen projection data sent by the mobile terminal in real time.

When a user performs mirror image screen projection on a mobile terminal and a display device, the mobile terminal and the display device are usually required to be connected in a local area network formed by the same router in advance. In the mirror image screen projection process, the mobile terminal records the screen of the content displayed by the mobile terminal in real time, and forwards the screen recording data to the display equipment through the router, and the display equipment decodes and displays in real time.

However, in the mirror image screen-throwing process of the mobile terminal and the display device, when the camera of the mobile terminal is not started, the screen-throwing data displayed in the display device is also switched along with the switching of the horizontal screen and the vertical screen of the mobile terminal. However, when the camera of the mobile terminal is turned on, no matter the mobile terminal is in a horizontal screen state or a vertical screen state, the captured mobile phone screen is in a vertical screen, the camera screen coming out of the camera is in an upward direction, and for the scene, when the mobile terminal is in the vertical screen state, the screen throwing data displayed at the display equipment end is also in a vertical screen state, namely, the display equipment end is in a normally playing state, however, when the mobile terminal is turned on, when the mobile terminal is in the horizontal screen state, the display equipment cannot acquire whether the mobile terminal is in the horizontal screen state or the vertical screen state, so that the display equipment can display the screen in the vertical screen direction of the mobile phone, and the captured camera screen is in a state that a scene lies down, so that the user experience feel bad is caused.

Disclosure of Invention

Some embodiments of the present application provide a mobile terminal, a display device, and a method for adjusting a screen projection direction, so as to solve the problem that when a camera of the mobile terminal is in an on state, the display device cannot acquire a switching state of the mobile terminal, so that screen projection data displayed by the display device does not conform to user watching experience and user experience feel is poor.

In a first aspect, some embodiments of the present application provide a mobile terminal including a camera, a display unit, a communication unit, a pose sensor, and a processor. Wherein the display unit is configured to display a terminal interface; the communication unit is configured to establish a screen-throwing connection with the display device; and the pose sensor is configured to detect pose information of the mobile terminal. The processor is configured to perform the following program steps:

generating screen projection data and transmitting the screen projection data to a display device;

responding to a camera opening event, and detecting direction information of the mobile terminal relative to a vertical plane according to the pose information;

and sending the direction information to the display equipment so that the display equipment displays the screen throwing data according to the direction information.

Based on the mobile terminal, the first aspect of the present application further provides a method for adjusting a screen projection direction, where the method includes the following steps:

Generating screen projection data and transmitting the screen projection data to a display device;

responding to a camera opening event, and detecting direction information of the mobile terminal relative to a vertical plane according to the pose information;

and sending the direction information to the display equipment so that the display equipment displays the screen throwing data according to the direction information.

According to the technical scheme, the mobile terminal and the method for adjusting the screen projection direction can generate screen projection data and send the screen projection data to the display device. And responding to a camera opening event, detecting the direction information of the mobile terminal relative to the vertical plane according to the pose information, and sending the direction information to the display equipment so that the display equipment displays the screen throwing data according to the direction information. After receiving the direction information, the display device can acquire the transverse and vertical screen states or the placement angles of the mobile terminal by analyzing the direction information, and rotate the picture display control according to the angle information. Therefore, the display of the screen throwing data accords with the watching experience of the user, and the experience of the user is further improved.

In a second aspect, some embodiments of the present application also provide a display device including a display, a communicator, and a controller. Wherein the display is configured to display a mirrored screen-drop interface; the communicator is configured to establish a screen-drop connection with the mobile terminal. The controller is configured to perform the following program steps:

Receiving screen throwing data and direction information sent by a mobile terminal; the direction information is the direction information of the mobile terminal relative to the vertical plane;

analyzing the direction information to obtain an angle value of the mobile terminal relative to a vertical plane;

and rotating a picture display control of the display equipment according to the angle value, and displaying the screen throwing data according to the rotated picture display control.

Based on the display device, the second aspect of the application further provides a method for adjusting the direction of a screen projection picture, which comprises the following steps:

receiving screen throwing data and direction information sent by a mobile terminal; the direction information is the direction information of the mobile terminal relative to the vertical plane;

analyzing the direction information to obtain an angle value of the mobile terminal relative to a vertical plane;

and rotating a picture display control of the display equipment according to the angle value, and displaying the screen throwing data according to the rotated picture display control.

According to the technical scheme, the display device and the method for adjusting the screen projection image direction can obtain the direction information such as the horizontal and vertical screen states or the placement angles of the mobile terminal, analyze the direction information to obtain the angle value of the mobile terminal relative to the vertical plane, rotate the image display control of the display device according to the angle value, display the screen projection data according to the rotated image display control, and therefore display the screen projection data according to the user's viewing experience can be guaranteed, and the user experience is further improved.

In a third aspect, some embodiments of the present application further provide a mobile terminal, including a camera, a display unit, a communication unit, a pose sensor, and a processor. Wherein the display unit is configured to display a terminal interface; the communication unit is configured to establish a screen-throwing connection with the display device; the pose sensor is configured to detect pose information of the mobile terminal. The processor is configured to perform the following program steps:

responding to a camera opening event, and controlling the pose sensor to detect an angle value of the mobile terminal relative to a vertical plane;

capturing a content image currently displayed by the display unit;

rotating the content image according to the angle value;

performing media asset data encoding based on the rotated content image to generate screen projection data;

and sending the screen projection data to a display device so that the display device decodes and displays the screen projection data.

Based on the mobile terminal of the third aspect, some embodiments of the present application further provide a display device including a display, a communicator, and a controller. Wherein the display is configured to display a mirrored screen-drop interface; the communicator is configured to establish a screen-drop connection with the mobile terminal. The controller is configured to perform the following program steps:

Receiving screen throwing data sent by a mobile terminal, wherein the screen throwing data is obtained after a content image rotates according to an angle value;

controlling the display to display the screen throwing data,

according to the technical scheme, the mobile terminal provided by the third aspect of the application responds to the camera opening event, and the pose sensor is controlled to detect the angle value of the mobile terminal relative to the vertical plane; capturing a content image currently displayed by a display unit; rotating the content image according to the angle value; performing media asset data encoding based on the rotated content image to generate screen projection data; and transmitting the screen projection data to the display device so that the display device decodes and displays the screen projection data. The mobile terminal can perform angle rotation and other processing on the screen throwing data at the data source end without performing rotation angle and other processing on the display equipment end, the display equipment can directly apply the screen throwing data sent by the mobile terminal, and the screen throwing data can be displayed after simple decoding, so that the efficiency of the display equipment in displaying the screen throwing data is improved, and meanwhile, the watching experience of a user is improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic diagram of an operation scenario between a display device and a control device provided in some embodiments of the present application;

FIG. 2 is a block diagram of a hardware configuration of the display device of FIG. 1 provided in some embodiments of the present application;

FIG. 3 is a block diagram of a hardware configuration of the control device of FIG. 1 provided in some embodiments of the present application;

FIG. 4 is a schematic diagram of a software configuration of the display device of FIG. 1 according to some embodiments of the present application;

fig. 5 is a schematic view of a screen projection effect of a mobile terminal in a vertical screen in a non-camera application scenario provided in some embodiments of the present application;

fig. 6 is a schematic view of a screen projection effect of a mobile terminal as a horizontal screen in a non-camera application scenario provided in some embodiments of the present application;

fig. 7 (a) is a schematic diagram of a picture captured when a mobile terminal is a vertical screen in a camera application scenario provided in some embodiments of the present application;

fig. 7 (b) is a schematic diagram of a picture captured when a mobile terminal is a horizontal screen in a camera application scenario provided in some embodiments of the present application;

fig. 8 (a) is a schematic diagram of a screen projection data display effect when a mobile terminal is a vertical screen in a camera application scenario provided in some embodiments of the present application;

fig. 8 (b) is a schematic diagram of a screen projection data display effect when a mobile terminal is a horizontal screen in a camera application scenario provided in some embodiments of the present application;

Fig. 9 is a flowchart illustrating a method for adjusting a screen projection direction according to some embodiments of the present application;

fig. 10 (a) is a schematic diagram of an effect of 0 degrees of the mobile terminal with respect to a vertical plane according to some embodiments of the present application;

fig. 10 (b) is a schematic diagram of an effect of 90 degrees of the mobile terminal with respect to a vertical plane according to some embodiments of the present application;

fig. 10 (c) is a schematic diagram illustrating an effect of 180 degrees of the mobile terminal with respect to a vertical plane according to some embodiments of the present application;

fig. 10 (d) is a schematic diagram of an effect of 270 degrees of the mobile terminal provided in some embodiments of the present application with respect to a vertical plane;

fig. 11 is a schematic flow chart of detecting direction information of a mobile terminal relative to a vertical plane according to some embodiments of the present application;

fig. 12 is a schematic flow chart of displaying screen projection data according to direction information by a display device according to some embodiments of the present application;

fig. 13 is a schematic diagram of an effect of a display device according to some embodiments of the present application to display screen projection data according to direction information;

fig. 14 is a schematic diagram of a mobile terminal monitoring flow after a camera application provided in some embodiments of the present application exits;

fig. 15 is a schematic flow chart of generating and processing screen-projection data by a mobile terminal according to some embodiments of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of some embodiments of the present application more clear, the technical solutions of some embodiments of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application.

It should be noted that the brief description of the terms in some embodiments of the present application is only for convenience in understanding the embodiments described below, and is not intended to limit the implementation of some embodiments of the present application. Unless otherwise indicated, these terms should be construed in their ordinary and customary meaning.

The terms first, second, third and the like in the description and in the claims and in the above-described figures are used for distinguishing between similar or similar objects or entities and not necessarily for describing a particular sequential or chronological order, unless otherwise indicated. It is to be understood that the terms so used are interchangeable under appropriate circumstances.

The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or apparatus that comprises a list of elements is not necessarily limited to all elements explicitly listed, but may include other elements not expressly listed or inherent to such product or apparatus.

The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware or/and software code that is capable of performing the function associated with that element.

Fig. 1 is a schematic diagram of an operation scenario between a display device and a control device according to some embodiments of the present application. As shown in fig. 1, a user may operate the display device 200 through the mobile terminal 300 and the control device 100.

In some embodiments, the mobile terminal 300 may install a software application with the display device 200, implement connection communication through a network communication protocol, and achieve the purpose of one-to-one control operation and data communication. The audio/video content displayed on the mobile terminal 300 can also be transmitted to the display device 200, so as to realize the synchronous display function.

As also shown in fig. 1, the display device 200 is also in data communication with the server 400 via a variety of communication means. The display device 200 may be permitted to communicate via a USB communication connection, a Local Area Network (LAN), a Wireless Local Area Network (WLAN), and other networks.

The display apparatus 200 may additionally provide a smart network television function of a computer support function, including, but not limited to, a network television, a smart television, an Internet Protocol Television (IPTV), etc., in addition to the broadcast receiving television function.

Fig. 2 is a block diagram of a hardware configuration of the display device 200 of fig. 1 provided in some embodiments of the present application.

In some embodiments, display apparatus 200 includes at least one of a modem 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, memory, a power supply, a user interface.

In some embodiments, the detector 230 is used to collect signals of the external environment or interaction with the outside.

In some embodiments, the display 260 includes a display screen component for presenting a picture, and a driving component for driving an image display, for receiving an image signal from the controller output, for displaying video content, image content, and components of a menu manipulation interface, and a user manipulation UI interface, etc.

In some embodiments, communicator 220 is a component for communicating with external devices or servers 400 according to various communication protocol types.

In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored on the memory. The controller 250 controls the overall operation of the display apparatus 200.

In some embodiments, a user may input a user command through a Graphical User Interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the Graphical User Interface (GUI).

In some embodiments, user interface 280 is an interface that may be used to receive control inputs.

Fig. 3 is a block diagram of a hardware configuration of the control device in fig. 1 according to some embodiments of the present application. As shown in fig. 3, the control device 100 includes a controller 110, a communication interface 130, a user input/output interface, a memory, and a power supply.

The control device 100 is configured to control the display device 200, and can receive an input operation instruction of a user, and convert the operation instruction into an instruction recognizable and responsive to the display device 200, functioning as an interaction between the user and the display device 200.

In some embodiments, the control device 100 may be a smart device. Such as: the control apparatus 100 may install various applications for controlling the display apparatus 200 according to user's needs.

In some embodiments, as shown in fig. 1, a mobile terminal 300 or other intelligent electronic device may function similarly to the control device 100 after installing an application that manipulates the display device 200.

The controller 110 includes a processor unit 112 and RAM 113 and ROM 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation and operation of the control device 100, as well as the communication collaboration among the internal components and the external and internal data processing functions.

The communication interface 130 enables communication of control signals and data signals with the display device 200 under the control of the controller 110. The communication interface 130 may include at least one of a WiFi chip 131, a bluetooth module 132, an NFC module 133, and other near field communication modules.

A user input/output interface 140, wherein the input interface includes at least one of a microphone 141, a touchpad 142, a sensor 143, keys 144, and other input interfaces.

In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input-output interface 140. The control device 100 is provided with a communication interface 130 such as: the WiFi, bluetooth, NFC, etc. modules may send the user input instruction to the display device 200 through a WiFi protocol, or a bluetooth protocol, or an NFC protocol code.

A memory 190 for storing various operation programs, data and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 may store various control signal instructions input by a user.

A power supply 180 for providing operating power support for the various elements of the control device 100 under the control of the controller.

Fig. 4 is a schematic view of a software configuration of the display device in fig. 1 according to some embodiments of the present application, in some embodiments, the system is divided into four layers, namely, an application layer (application layer), an application framework layer (Application Framework layer), a system library layer (system runtime layer), and a kernel layer from top to bottom.

In some embodiments, at least one application program is running in the application program layer, and these application programs may be a Window (Window) program of an operating system, a system setting program, a clock program, a camera application, and the like; or may be an application developed by a third party developer.

The framework layer provides an application programming interface (Aplication Pogramming Iterface, API) and programming framework for application programs of the application layer. The application framework layer includes a number of predefined functions. The application framework layer corresponds to a processing center that decides to let the applications in the application layer act.

As shown in fig. 4, the application framework layer in some embodiments of the present application includes a manager (manager), a Content Provider (Content Provider), a View System (View System), and the like. Wherein the manager comprises at least one of the following modules: an Activity Manager (Activity Manager) is used to interact with all activities that are running in the system; a Location Manager (Location Manager) is used to provide system services or applications with access to system Location services; a Package Manager (Package Manager) for retrieving various information about an application Package currently installed on the device; a notification manager (Notification Manager) for controlling the display and clearing of notification messages; a Window Manager (Window Manager) is used to manage bracketing icons, windows, toolbars, wallpaper, and desktop components on the user interface.

In some embodiments, the activity manager is to: managing the lifecycle of the individual applications and typically the navigation rollback functionality.

In some embodiments, a window manager is used to manage all window programs.

In some embodiments, the system runtime layer provides support for the upper layers, namely the framework layer, and when the framework layer is accessed, the operating system runs the C/C++ libraries contained in the system runtime layer to implement the functions to be implemented by the framework layer.

In some embodiments, the kernel layer is a layer between hardware and software. As shown in fig. 4, the kernel layer contains at least one of the following drivers: audio drive, display drive, bluetooth drive, camera drive, WIFI drive, USB drive, HDMI drive, sensor drive (e.g., fingerprint sensor, temperature sensor, touch sensor, pressure sensor, etc.), and the like.

In some embodiments, the kernel layer further includes a power driver module for power management.

In some embodiments, the software programs and/or modules corresponding to the software architecture in fig. 4 are stored in the first memory or the second memory shown in fig. 2 or fig. 3.

The mobile terminal 300 and the display device 200 may establish a screen connection, so that the display device 200 may synchronously play the screen content displayed by the mobile terminal 300. In this embodiment, when the mobile terminal 300 is connected to the display device 200 in a screen-projection manner, a specific screen-projection connection protocol may be used. For example, the present application is not particularly limited, and other network protocols may be self-developed, such as a connection protocol such as Miracast, airPlay, digital living network alliance (Digital Living Network Alliance, DLNA), universal serial bus (Universal Serial Bus, USB), or the like.

When the mobile terminal 300 performs a screen-casting connection with the display device 200, the mobile terminal 300 serves as a data source terminal for providing screen-casting data, and the display device 200 serves as a receiving terminal for receiving and displaying the screen-casting data. Taking the mobile terminal 300 as the mobile phone device a and the display device 200 as the television device B as an example, in some embodiments, the screen-casting connection between the mobile phone device a and the television device B may be implemented as follows.

For example, the mobile phone device a is used as a data source end, the television device B is used as a screen-throwing display end, and after the mobile phone device a and the television device B are communicated through a USB connection protocol or a self-development protocol and are successfully connected through screen throwing, the mobile phone device a starts recording a screen and generates an original video frame, for example, the original video frame may be a data frame of h264 or h 265. After generating the video frame, the mobile phone device a may send to the television device B through a network or a USB transmission protocol. After receiving the video frame, the television apparatus B may decode the original video frame data by a video decoder to form screen projection data, and control the display 260 to display the screen projection data. For example, in some embodiments, the drop data may be displayed through a visual display control, such as SufaceView or textureView.

The mobile terminal 300 may be divided into a camera open/close state and a camera open state in an actual screen-projection use scenario. It can be understood that the camera off state is a screen projection event not in the camera application scene, and the camera on state is a screen projection event in the camera application scene. The following describes the above two scenarios in detail with reference to the accompanying drawings.

Fig. 5 is a schematic view of a screen projection effect of a mobile terminal for a vertical screen in a non-camera application scenario provided by some embodiments of the present application, and fig. 6 is a schematic view of a screen projection effect of a mobile terminal for a horizontal screen in a non-camera application scenario provided by some embodiments of the present application. Referring to fig. 5 and fig. 6, when the mobile terminal 300 is not in the camera application scene, the camera is not turned on, and when the mobile terminal 300 switches between the vertical screen and the horizontal screen, the application in the mobile terminal 300 also performs rotation of the synchronous screen.

In some embodiments, when the camera of the mobile terminal 300 is not turned on, the mobile terminal 300 may monitor the landscape screen state and the portrait screen state of the mobile terminal 300 through a callback function. The mobile terminal 300 may monitor through an interface of an onconfigured changed callback function, and when the mobile terminal 300 has an initial state, it may feed back whether the mobile terminal 300 is in a horizontal screen state or a vertical screen state through the callback function after monitoring that the state changes, so that the mobile terminal 300 may adaptively adjust the change of the screen recording parameter according to the state in which the mobile terminal 300 is located, so that the recorded screen may be consistent with the direction of the mobile terminal 300.

For example, in some embodiments, when the mobile terminal 300 is in a portrait state, the recording may be encoded at a resolution of 1080×1920, and when the mobile terminal 300 is in a landscape state, the recording may be encoded at a resolution of 1920×1080. Assuming that the initial state of the mobile terminal 300 is a vertical screen state, it is encoded at a resolution of 1080×1920. When the vertical screen state is switched to the horizontal screen state, the callback function detects that the direction changes and feeds back the change state. After receiving the change to the landscape screen state, the mobile terminal 300 may adjust the resolution encoding format of 1080×1920 to the resolution encoding format of 1920×1080 and transmit the encoded screen input data to the display device 200. In this way, after receiving the adjusted screen projection data, the display device 200 may directly display the screen projection data in the direction consistent with the direction of the mobile terminal 300, so as to conform to the visual viewing effect of the user.

When the mobile terminal 300 is in the camera application scene, the camera is turned on, but due to reasons such as an interface or configuration, when the mobile terminal 300 switches between the vertical screen state and the horizontal screen state, the state change of the mobile terminal 300 cannot be fed back through a callback function, so that the resolution matching with the direction of the mobile terminal 300 cannot be adaptively adjusted by the screen recording application in the scene, and the problem that when the mobile terminal 300 is in the horizontal screen state, the screen throwing data displayed by the display terminal 200 does not accord with the viewing experience of a user may occur.

Fig. 7 (a) is a schematic diagram of a picture captured when a mobile terminal is a vertical screen in a camera application scene provided by some embodiments of the present application, and fig. 7 (b) is a schematic diagram of a picture captured when a mobile terminal is a horizontal screen in a camera application scene provided by some embodiments of the present application. When the camera of the mobile terminal 300 is turned on, the captured screen is a portrait screen regardless of whether the mobile terminal 300 is in a landscape screen state or a portrait screen state, as shown in fig. 7 (a) and 7 (b). Therefore, when the mobile terminal 300 is in a portrait state, it is recorded and encoded at a resolution of 1080×1920, and after being transmitted to the display device 200, the display device 200 decodes at this resolution, and the projection data displayed by the display device 200 is also portrait. Fig. 8 (a) is a schematic diagram of a display effect of screen projection data when the mobile terminal is in a vertical screen state in the application scenario of the camera according to some embodiments of the present application, as shown in fig. 8 (a), that is, when the mobile terminal 300 is in a vertical screen state, a direction of the screen projection data displayed on the display device 200 end accords with a user viewing experience.

In the case where the camera is turned on, when the mobile terminal 300 is in the landscape screen state, the content image captured by the camera does not rotate, and as shown in fig. 7 (b), the captured picture of the ship does not rotate, but is still in the portrait screen direction. However, as can be easily seen from fig. 7 (a), the camera application is rotated, and the original application modes such as night scenes, figures, photos, and videos of the camera are rotated, for example, when the mobile terminal 300 is horizontally placed right, it is rotated by 90 degrees, and when the mobile terminal 300 is horizontally placed left, it is rotated by 270 degrees, so that the resolution of the projection data recorded by the mobile terminal 300 is 1080×1920 in the vertical screen state. That is, in the case where the camera is turned on, the display device 200 cannot learn the real state of the mobile terminal 300, and when the mobile terminal 300 is in the landscape state, recording encoding is still performed according to the resolution of 1080×1920, and the recording encoding is sent to the display device 200, so that the screen projection data displayed by the display device 200 does not conform to the viewing experience of the user.

As shown in fig. 8 (b), when the mobile terminal 300 is switched from the vertical screen state to the horizontal screen state in the screen-throwing process of the display device 200, the state change event of the mobile terminal 300 cannot be known, so that even when the mobile terminal 300 is in the horizontal screen state, recording and encoding cannot be performed according to the horizontal screen resolution, and therefore, the problem that the screen-throwing data displayed by the display device 200 does not conform to the viewing experience of the user may be caused, resulting in poor user experience.

In order to solve the problem that when the camera of the mobile terminal 300 is in an on state, the display device 200 cannot acquire the direction switching state of the mobile terminal 300, so that the screen projection data displayed by the display device 200 does not accord with the watching experience of the user and the experience of the user is poor, some embodiments of the present application provide a mobile terminal 300. The mobile terminal 300 includes a camera 310, a display unit 320, a communication unit 330, a pose sensor 340, and a processor 350. Wherein the display unit 320 is configured to display a terminal interface; the communication unit 330 is configured to establish a screen-drop connection with the display device 200; the pose sensor 340 is configured to detect pose information of the mobile terminal 300. The present application aims to calculate, when the camera 310 of the mobile terminal 300 is turned on, direction information of the mobile terminal 300 with respect to a vertical plane through the pose sensor 340, and send the direction information to the display device 200, so that the display device 200 displays screen projection data according to the direction information.

In order to facilitate understanding of the technical solutions in some embodiments of the present application, the following details of each step are described with reference to some specific embodiments and the accompanying drawings. Fig. 9 is a flowchart of a method for adjusting a screen orientation according to some embodiments of the present application, as shown in fig. 9, the processor 350 is configured to execute the following steps S1-S3:

step S1, the mobile terminal 300 generates screen projection data and transmits the screen projection data to the display device 200.

Before the mobile terminal 300 generates the screen-shot data, a screen-shot connection protocol for establishing a communication connection between the mobile terminal 300 and the display device 200 needs to be established. After the screen-cast connection protocol is established successfully, the mobile terminal 300 records the screen data and encodes the screen data to generate the screen-cast data. After the screen data is generated, the mobile terminal 300 may transmit the screen data to the display device 200, so that the display device 200 may display the screen data. After the completion of the execution of step S1, the following step S2 may be executed.

Step S2, in response to the starting event of the camera 310, the direction information of the mobile terminal 300 relative to the vertical plane is detected according to the pose information.

Fig. 10 (a) is a schematic view of the effect of the mobile terminal provided in some embodiments of the present application, where, as shown in fig. 10 (a), in some embodiments, the initial direction of the mobile terminal 300 may be defaulted to a vertical plane, and when the pose of the mobile terminal changes, the vertical plane may be taken as a reference, and the changed angle may be taken as changed direction information. Referring to fig. 10 (b), 10 (c), and fig. 10 (d), fig. 10 (b) is a schematic diagram illustrating an effect that the mobile terminal provided in some embodiments of the present application is 90 degrees with respect to a vertical plane, that is, the mobile terminal 300 rotates 90 degrees with respect to an initial vertical plane; fig. 10 (c) is a schematic diagram illustrating an effect of 180 degrees of the mobile terminal with respect to the vertical plane according to some embodiments of the present application, that is, 180 degrees of rotation of the mobile terminal 300 with respect to the initial vertical plane; fig. 10 (d) is a schematic diagram illustrating an effect that the mobile terminal provided in some embodiments of the present application is 270 degrees with respect to the vertical plane, that is, the mobile terminal 300 rotates 270 degrees with respect to the initial vertical plane.

It may be appreciated that when the camera 310 is not turned on, the mobile terminal 300 may monitor the change of state through the interface of the callback function when the mobile terminal 300 switches between the portrait screen and the landscape screen. When the state of the mobile terminal 300 is monitored to change, the callback function can be used for feeding back whether the mobile terminal 300 is in a horizontal screen state or a vertical screen state, so that the mobile terminal 300 can adaptively adjust the change of the screen recording parameters according to the state of the mobile terminal 300, and the recorded screen picture can be consistent with the direction of the mobile terminal 300. In this way, after receiving the screen projection data sent by the mobile terminal 300, the display device 200 also displays the screen projection data in the same direction as the mobile terminal 300, so as to meet the viewing experience of the user.

In order for the screen shot data displayed by the display device 200 when the camera 310 is turned on to also conform to the viewing experience of the user, in some embodiments, in response to an on event of the camera 310, orientation information of the mobile terminal 300 with respect to a vertical plane may be detected according to pose information. Fig. 11 is a schematic flow chart of detecting direction information of a mobile terminal relative to a vertical plane according to some embodiments of the present application, as shown in fig. 11, the mobile terminal 300 may register a listening interface of the pose sensor 340 first, and after the listening interface is registered successfully, the pose sensor 340 may be controlled to monitor the direction of the mobile terminal 300 through the listening interface, so as to detect the direction information of the mobile terminal 300 relative to the vertical plane according to the pose information and the listening interface.

In the embodiment of the present application, the pose information is detected by the pose sensor 340, and the pose sensor 340 is a sensor capable of sensing the position and the pose of the object to be measured and converting the position and the pose into the available output signal. In some embodiments, the pose sensor 340 may be a direction sensor or a geomagnetic sensor, or may be another type of sensor capable of acquiring direction information, which is not specifically limited in this application.

In the camera application scenario, when the pose sensor 340 detects pose information of the mobile terminal 300, the mobile terminal 300 may register a monitoring interface of the pose sensor 340 first, so that the mobile terminal 300 may acquire, through the monitoring interface, direction information returned by the pose sensor 340, and acquire whether the mobile terminal 300 is in a horizontal screen state or a vertical screen state, and what the corresponding rotation angle is. In some embodiments, the interaction instruction may be generated by using information such as a horizontal and vertical screen state and a rotation angle, so that the mobile terminal 300 may send the interaction instruction including information such as a horizontal and vertical screen state and a rotation angle to the display device 200, so that the display device 200 performs a corresponding function according to the interaction instruction. For example, the interaction instruction may be "phone_orientation:90", where phone may represent the mobile terminal 300, orientation may represent the direction of the mobile terminal 300 relative to the vertical plane, and the interaction instruction may represent the meaning that the rotation angle of the mobile terminal 300 relative to the vertical plane is 90 degrees currently, and is in a landscape screen placement state.

For example, after the mobile terminal 300 determines that the camera application is opened, the monitoring interface of the pose sensor 340 may be registered through a preset instruction, for example, the monitoring interface may be albuminorientioneventlistener, so that the mobile terminal 300 may control the pose sensor 340 to monitor the direction of the mobile terminal 300 through the monitoring interface, so as to detect the direction information of the mobile terminal 300 relative to the vertical plane according to the pose information and the monitoring interface. It can be understood that the direction information includes information such as a horizontal/vertical screen state and a rotation angle contained in the interaction instruction. After the completion of the step S2, the following step S3 may be performed.

Step S3, the mobile terminal 300 transmits the direction information to the display device 200, so that the display device 200 displays the screen-casting data according to the direction information.

After generating the direction information, the mobile terminal 300 may transmit the direction information to the display device 200 through a network protocol or a USB transfer protocol established between the mobile terminal 300 and the display device 200, or the like. Thus, the display device 200 may display the screen projection data according to the direction information after receiving the screen projection data and the direction information transmitted from the mobile terminal 300.

Fig. 12 is a schematic flow chart of displaying screen projection data according to direction information by the display device provided in some embodiments of the present application, as shown in fig. 12, in order to display the screen projection data sent by the mobile terminal 300 according to the direction information, after receiving the screen projection data sent by the mobile terminal 300 and the direction information of the mobile terminal 300 relative to the vertical plane, the display device 200 may analyze the direction information to obtain an angle value of the mobile terminal 300 relative to the vertical plane, and finally rotate a screen display control of the display device 200 according to the angle value, and display the screen projection data according to the rotated screen display control.

In order to display the screen-casting data, in some embodiments, the display device 200 may first establish a transmission channel for transmitting the screen-casting data, decode the received screen-casting data after the transmission channel is established, receive the decoded screen-casting data through the transmission channel, and finally display the decoded screen-casting data through a display control such as a SufaceView or TextureView.

Fig. 13 is a schematic diagram showing the effect of displaying projection data by the display device according to the direction information according to some embodiments of the present application, as shown in fig. 13, when the camera application is opened, the mobile terminal 300 may calculate the direction information relative to the vertical plane by using the pose sensor 340, and send the direction information to the display device 200 through a network connection protocol or a USB protocol established with the display device 200, so as to notify the display device 200 of the direction of the mobile terminal 300 at this time. After receiving the direction information, the display device 200 may obtain the landscape screen state or the landscape angle of the mobile terminal 300 by analyzing the direction information, and rotate the screen display control according to the angle information. For example, when the mobile terminal 300 is in the landscape state and the rotation angle is 90, the display device 200 may rotate the screen display control by 90 degrees, so that the interface for displaying the projection data is also in the landscape state shown in fig. 3, and thus, the display of the projection data accords with the viewing experience of the user, and further, the experience of the user is improved.

As can be seen from the above technical solutions, the mobile terminal 300 in the above embodiments can generate the screen projection data and send the screen projection data to the display device 200. In response to the camera 310 on event, direction information of the mobile terminal 300 with respect to the vertical plane is detected according to the pose information, and the direction information is transmitted to the display apparatus 200, so that the display apparatus 200 displays the projection data according to the direction information. After receiving the direction information, the display device 200 may obtain the landscape screen state or the landscape angle of the mobile terminal 300 by analyzing the direction information, and rotate the screen display control according to the angle information. Therefore, the display of the screen throwing data accords with the watching experience of the user, and the experience of the user is further improved.

In some embodiments, the interface listening function of the pose sensor 340 may be canceled when the mobile terminal 300 exits the camera application. Fig. 14 is a schematic diagram of a mobile terminal monitoring flow after the exit of the camera application according to some embodiments of the present application, as shown in fig. 14, in response to a closing event of the camera 310, the mobile terminal 300 may monitor a screen state of the mobile terminal 300 in a screen throwing process, where the screen state may include a horizontal screen state and a vertical screen state. In the step of performing the monitoring of the screen state of the mobile terminal 300 during the screen-throwing process, as described in the foregoing embodiment, in the non-camera application scenario, that is, when the camera 310 is turned off, a callback function for monitoring the screen state change is set, and in response to detecting that the screen state of the mobile terminal 300 changes, the screen state of the mobile terminal 300 is fed back to the mobile terminal 300 through the callback function. Thus, the screen state of the mobile terminal 300 can be acquired.

Upon detection, if the screen state of the mobile terminal 300 is a landscape screen state, the screen data may be encoded according to the first resolution to generate first screen projection data and the first screen projection data may be transmitted to the display device 200. For example, when the screen state of the mobile terminal 300 is a landscape screen state, the first resolution may be 1920×1080 resolution. If the screen state of the mobile terminal 300 is the portrait state, the screen data is encoded according to the second resolution to generate second screen projection data, and the second screen projection data is transmitted to the display device 200. For example, when the screen state of the mobile terminal 300 is a portrait state, the second resolution may be 1080×1920 resolution. In this way, when the mobile terminal 300 exits from the application scene of the camera, the function of monitoring the screen state thereof through the callback function can be automatically switched, and it can be understood that when the camera 310 is opened, the function of monitoring the monitoring interface of the pose sensor 340 is switched, so that when the mobile terminal 300 performs the horizontal-vertical screen switching in any scene, the screen throwing data displayed on the display device 200 end accords with the watching experience of the user, and further the user experience is improved.

Based on the above mobile terminal 300, some embodiments of the present application further provide a method for adjusting a screen direction, where the method may be applied to the mobile terminal 300 in the above embodiments, and the mobile terminal 300 includes a camera 310, a display unit 320, a communication unit 330, a pose sensor 340, and a processor 350, and the method for adjusting a screen direction may include the following:

the mobile terminal 300 generates screen projection data and transmits the screen projection data to the display device 200; in response to the camera 310 on event, detecting direction information of the mobile terminal 300 relative to the vertical plane according to the pose information; the direction information is transmitted to the display apparatus 200 so that the display apparatus 200 displays the screen-casting data according to the direction information.

As can be seen from the above technical solutions, the method for adjusting the screen projection direction in the above embodiments can generate the screen projection data and send the screen projection data to the display device 200. In response to the camera 310 on event, direction information of the mobile terminal 300 with respect to the vertical plane is detected according to the pose information, and the direction information is transmitted to the display apparatus 200, so that the display apparatus 200 displays the projection data according to the direction information. After receiving the direction information, the display device 200 may obtain the landscape screen state or the landscape angle of the mobile terminal 300 by analyzing the direction information, and rotate the screen display control according to the angle information. Therefore, the display of the screen throwing data accords with the watching experience of the user, and the experience of the user is further improved.

Some embodiments of the present application also provide a display device 200 including a display 260, a communicator 220, and a controller 250. Wherein the display 260 is configured to display a mirrored screen-drop interface; the communicator 220 is configured to establish a screen-drop connection with the mobile terminal 300, and the controller 250 is configured to perform the following procedure:

receiving screen projection data and direction information sent by the mobile terminal 300; the direction information is direction information of the mobile terminal 300 with respect to a vertical plane; analyzing the direction information to obtain an angle value of the mobile terminal 300 relative to the vertical plane; and rotating the picture display control of the display device 200 according to the angle value, and displaying the screen throwing data according to the rotated picture display control.

In addition, in the process of displaying the screen projection data, the display device 200 may establish a transmission channel of the screen projection data, and decode the received screen projection data; and receiving the decoded screen throwing data through a transmission channel, and displaying the decoded screen throwing data through a picture display control.

Based on the display device 200, some embodiments of the present application further provide a method for adjusting a screen projection direction, where the method is applied to the display device 200, and the display device 200 includes a display 260, a communicator 220, and a controller 250, and the method for adjusting a screen projection direction includes:

Receiving screen projection data and direction information sent by the mobile terminal 300; wherein the direction information is direction information of the mobile terminal 300 with respect to a vertical plane; analyzing the direction information to obtain an angle value of the mobile terminal 300 relative to the vertical plane; and rotating the picture display control of the display device 200 according to the angle value, and displaying the screen throwing data according to the rotated picture display control.

As can be seen from the above technical solution, the display device 200 and the method for adjusting the screen-throwing direction provided in the foregoing embodiments may obtain direction information such as a horizontal-vertical screen state or a placement angle of the mobile terminal 300, and analyze the direction information to obtain an angle value of the mobile terminal 300 relative to a vertical plane, rotate a screen display control of the display device 200 according to the angle value, and display screen-throwing data according to the rotated screen display control, so that display of the screen-throwing data can be ensured to conform to viewing experience of a user, and further experience of the user is improved.

Some embodiments of the present application also provide another mobile terminal 300, including a camera 310, a display unit 320, a communication unit 330, a pose sensor 340, and a processor 350. Wherein the display unit 320 is configured to display a terminal interface, the communication unit 330 is configured to establish a screen-casting connection with the display device 200, and the pose sensor 340 is configured to detect pose information of the mobile terminal 300. Fig. 15 is a flowchart illustrating a mobile terminal generating and processing screen-projection data according to some embodiments of the present application, as shown in fig. 15, where the processor 350 in the mobile terminal 300 is configured to execute the following steps S10-S50:

Step S10: in response to the camera 310 on event, the pose sensor 340 is controlled to detect an angle value of the mobile terminal 300 with respect to a vertical plane.

In some embodiments, the mobile terminal 300 may also register a listening interface of the pose sensor 340, and control the pose sensor 340 to listen for direction information of the mobile terminal 300 relative to the vertical plane through the listening interface, and the detailed process may refer to the foregoing embodiments and will not be repeated here. The angle value of the mobile terminal 300 with respect to the vertical plane may also be detected in other forms, which is not particularly limited in this application.

Step S20: the mobile terminal 300 captures a content image currently displayed by the display unit 320.

The display unit 320 is configured to display a terminal interface, when the camera 310 is turned on, the display unit 320 displays a content image within a shooting range of the camera, and the mobile terminal 300 may capture the content image currently displayed by the display unit 320 to prepare data for the projection data.

Step S30: the mobile terminal 300 rotates the content image according to the detected angle value.

In order to facilitate the use of the display device 200, in some embodiments, the mobile terminal 300 may rotate the content image according to the detected angle value, so that the mobile terminal 300 may process the content image at the data source end, and the display device end does not need to process other processes, such as rotation angle, on the content image, which only needs simple decoding, so that the application of the user on the display device 200 end may be facilitated.

Step S40: the mobile terminal 300 performs media data encoding based on the rotated content image to generate projection data.

After the mobile terminal 300 rotates the content image at the data source end, in order to generate the screen projection data to which the display device 200 can be directly applied, the mobile terminal 300 may perform media data encoding based on the rotated content image to generate the screen projection data. It should be noted that, the screen projection data that the display device 200 can directly apply here refers to screen projection data that does not need to be processed by rotation of the display device 200, so that the efficiency of displaying the screen projection data by the display device 200 can be improved.

Step S50: the mobile terminal 300 transmits the screen-cast data to the display device 200 to cause the display device 200 to decode and display the screen-cast data.

After the mobile terminal 300 generates the screen projection data, the screen projection data can be sent to the display device 200, so that the display device 200 can display the screen projection data by directly decoding the screen projection data, and operations such as analyzing direction information and rotating a picture display control on the display device 200 are not required.

Based on the above mobile terminal 300, some embodiments of the present application further provide a display device 200, including a display 260, a communicator 220, and a controller 250. Wherein the display 260 is configured to display a mirrored screen-drop interface; the communicator 220 is configured to establish a screen-drop connection with the mobile terminal. The controller 250 is configured to receive the screen projection data transmitted from the mobile terminal 300 and control the display 260 to display the screen projection data, wherein the screen projection data is obtained by rotating the content image according to the angle value.

For example, the currently displayed content image captured by the mobile terminal 300 may be a screen original color-coded data YUV image, hereinafter referred to as YUV image. Among them, YUV is a color coding method. In some embodiments, Y may represent brightness luminence or Luma, i.e. gray scale values, and U and V represent chromaticity chromance or Chroma, in which, in the embodiments of the present application, the screen raw color-coded data YUV image may be understood as the content image currently displayed by the display unit 320.

After the mobile terminal 300 monitors the angle value relative to the vertical plane through the monitoring interface of the pose sensor 340, the captured YUV image may be video-encoded, and may be changed into stream data of h264 or h265, for example, and then may be sent to the display device 200 through a data transmission channel of a network connection or a USB connection established between the mobile terminal 300 and the display device 200. In this way, the display device 200, upon receiving the video stream data, decodes in the usual manner without rotating the picture display control.

As can be seen from the above technical solution, the mobile terminal 300 provided in the above embodiment responds to the camera opening event, and controls the pose sensor 340 to detect the angle value of the mobile terminal 300 relative to the vertical plane; capturing a content image currently displayed by the display unit 320; rotating the content image according to the angle value; performing media asset data encoding based on the rotated content image to generate screen projection data; and transmits the screen projection data to the display apparatus 200 so that the display apparatus 200 decodes and displays the screen projection data. The mobile terminal 300 can perform angle rotation and other processing on the screen projection data at the data source end, and the display device 200 end does not need to perform rotation angle and other processing, the display device 200 can directly apply the screen projection data sent by the mobile terminal 300, and the screen projection data can be displayed after simple decoding, so that the efficiency of displaying the screen projection data by the display device 200 is improved, and meanwhile, the viewing experience of a user can be improved.

The same and similar parts of the embodiments in this specification are referred to each other, and are not described herein.

It will be apparent to those skilled in the art that the techniques of embodiments of the present invention may be implemented in software plus a necessary general purpose hardware platform. Based on such understanding, the technical solutions in the embodiments of the present invention may be embodied essentially or in parts contributing to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method of the embodiments or some parts of the embodiments of the present invention.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

The foregoing description, for purposes of explanation, has been presented in conjunction with specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed above. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles and the practical application, to thereby enable others skilled in the art to best utilize the embodiments and various embodiments with various modifications as are suited to the particular use contemplated.

Claims (10)

1. A mobile terminal, comprising:

a camera;

a display unit configured to display a terminal interface;

a communication unit configured to establish a screen-casting connection with the display device;

a pose sensor configured to detect pose information of the mobile terminal;

a processor configured to:

generating screen projection data and transmitting the screen projection data to a display device;

responding to a camera opening event, and detecting direction information of the mobile terminal relative to a vertical plane according to the pose information;

and sending the direction information to the display equipment so that the display equipment displays the screen throwing data according to the direction information.

2. The mobile terminal of claim 1, wherein the processor is further configured to:

registering a monitoring interface of the pose sensor;

and controlling the pose sensor to monitor the direction of the mobile terminal through the monitoring interface so as to detect the direction information of the mobile terminal relative to a vertical plane according to the pose information and the monitoring interface.

3. The mobile terminal of claim 1, wherein the processor is further configured to:

establishing a screen-throwing connection protocol, wherein the screen-throwing connection protocol is used for establishing communication connection between the mobile terminal and the display equipment;

recording screen data after the screen-throwing connection protocol is established successfully;

and encoding the screen data to generate screen throwing data.

4. The mobile terminal of claim 1, wherein the processor is further configured to:

responding to a camera closing event, and monitoring the screen state of the mobile terminal in the screen throwing process; the screen state comprises a horizontal screen state and a vertical screen state;

if the screen state is the horizontal screen state, encoding the screen data according to a first resolution to generate first screen projection data, and transmitting the first screen projection data to the display equipment;

And if the screen state is the vertical screen state, encoding the screen data according to a second resolution to generate second screen projection data, and transmitting the second screen projection data to the display equipment.

5. The mobile terminal of claim 4, wherein the processor performs the step of listening to a screen state of the mobile terminal during a screen casting process, and is further configured to:

setting a callback function for monitoring screen state change;

and responding to the detection of the change of the screen state, and feeding back the screen state of the mobile terminal to the mobile terminal through the callback function.

6. A display device, characterized by comprising:

a display configured to display a mirrored screen interface;

a communicator configured to establish a screen-drop connection with the mobile terminal;

a controller configured to:

receiving screen throwing data and direction information sent by a mobile terminal; the direction information is the direction information of the mobile terminal relative to the vertical plane;

analyzing the direction information to obtain an angle value of the mobile terminal relative to a vertical plane;

and rotating a picture display control of the display equipment according to the angle value, and displaying the screen throwing data according to the rotated picture display control.

7. The display device of claim 6, wherein the controller is further configured to:

establishing a transmission channel of screen throwing data;

decoding the screen throwing data;

and receiving the decoded screen throwing data through the transmission channel, and displaying the decoded screen throwing data through the picture display control.

8. A mobile terminal, comprising:

a camera head:

a display unit configured to display a terminal interface;

a communication unit configured to establish a screen-casting connection with the display device;

a pose sensor configured to detect pose information of the mobile terminal;

a processor configured to:

responding to a camera opening event, and controlling the pose sensor to detect an angle value of the mobile terminal relative to a vertical plane;

capturing a content image currently displayed by the display unit;

rotating the content image according to the angle value;

performing media asset data encoding based on the rotated content image to generate screen projection data;

and sending the screen projection data to a display device so that the display device decodes and displays the screen projection data.

9. The method for adjusting the screen throwing picture direction is applied to a mobile terminal, and the mobile terminal comprises a camera, a display unit, a communication unit, a pose sensor and a processor, and is characterized by comprising the following steps of:

Generating screen projection data and transmitting the screen projection data to a display device;

responding to a camera opening event, and detecting direction information of the mobile terminal relative to a vertical plane according to the pose information;

and sending the direction information to the display equipment so that the display equipment displays the screen throwing data according to the direction information.

10. The method for adjusting the screen projection direction is applied to display equipment, and the display equipment comprises a display, a communicator and a controller, and is characterized by comprising the following steps of:

receiving screen throwing data and direction information sent by a mobile terminal; the direction information is the direction information of the mobile terminal relative to the vertical plane;

analyzing the direction information to obtain an angle value of the mobile terminal relative to a vertical plane;

and rotating a picture display control of the display equipment according to the angle value, and displaying the screen throwing data according to the rotated picture display control.

Images