WO2022193475A1

WO2022193475A1 - Display device, method for receiving screen projection content, and screen projection method

Info

Publication number: WO2022193475A1
Application number: PCT/CN2021/103779
Authority: WO
Inventors: 张敬坤; 朱雪生; 刘美玉; 林乐; 宋子全; 王之奎; 于颜梅
Original assignee: 青岛海信传媒网络技术有限公司
Priority date: 2021-03-19
Filing date: 2021-06-30
Publication date: 2022-09-22

Abstract

A display device, a method for receiving screen projection content, and a screen projection method. A controller of the display device is configured to: receive a screen projection media resource, and determine the attribute of the media resource; if the media resource attribute is video, detect the horizontal and vertical screen state of a display, acquire the aspect ratio of the video, and determine whether the aspect ratio of the video matches the horizontal and vertical screen state of the display; and if the media resource attribute is image, fill a screen according to the original proportion of the aspect ratio of the current image for displaying.

Description

Display device, method for receiving screencast content, and screencasting method

This application claims the priority of the Chinese patent application with the application number 202110297114.7 and the application title "A display device and screen projection method", which was submitted to the Chinese Patent Office on March 19, 2021, the entire contents of which are incorporated into this application by reference middle.

technical field

The present application relates to the technical field of smart TVs, and in particular, to a display device, a method for receiving screencast content, and a screencasting method.

Background technique

Screen sharing is the interactive operation of multiple display devices. Generally, a wireless network is used to transmit media resources, so that the content displayed on one display device can be synchronized to the second display device for the user to watch. Taking mobile phone screencasting as an example, for mobile phones and smart TVs connected to the same WiFi network, the screencasting operation command can be executed through the mobile phone to send the screen displayed on the mobile phone to the smart TV in the form of video stream, so as to use Big screen of smart TV for better user experience.

Since media assets can be vertical assets and landscape assets, when the media asset stream is synchronized on one device to a second display device, there is a possibility that the aspect ratio of the media asset may be different from the presentation method of the display device. , such as a media resource with a 4:3 display size and a display device with a 3:4 display size.

In order to fully display the screen on the mobile phone with the largest display ratio, the display device needs to adapt the media resources.

SUMMARY OF THE INVENTION

The present application provides a display device, a method for receiving screen-casting content, and a screen-casting method, so that the presentation state of the display is adapted to the screen-casting content.

In one aspect, the present application provides a display device, comprising: a display;

a rotation component, configured to drive the display to rotate, so that the display is in a rotation state of a landscape screen state or a portrait screen state;

a user interface, the communication is configured to connect to a terminal;

Controller, configured as:

Receive screen projection media resources, and determine the attributes of the media resources;

If the media resource attribute is video, detect the horizontal and vertical screen status of the display, obtain the video aspect ratio, and determine the matching situation between the video aspect ratio and the horizontal and vertical screen status of the display, and determine the matching situation according to the matching Circumstances determine whether the display needs to be rotated;

If the media resource attribute is a picture, the screen is filled and displayed according to the original aspect ratio of the current image.

In a second aspect, the present application provides a method for receiving screencast content, including:

In a third aspect, the present application provides a display device, including:

monitor;

a user interface, the communication is configured to connect to a terminal;

Controller, configured as:

receiving image information sent by the terminal, wherein, when the terminal is in a portrait mode, the image information includes valid information and left and right black information, and the valid information corresponds to the screen display content of the terminal;

If the current rotation state of the display does not match the portrait mode of the terminal, rotate the display to the portrait state;

The display is controlled to present a screen projection image based on the valid information, wherein the screen projection image is obtained by amplifying the valid information by a preset multiple.

In a fourth aspect, a screen projection method, applied to a display device, includes:

Description of drawings

In order to illustrate the technical solutions of the present application more clearly, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, for those of ordinary skill in the art, without creative work, the Additional drawings can be obtained from these drawings.

1 is an application scenario diagram of a display device of the present application;

Fig. 2 is the hardware configuration block diagram of the control device of the application;

3A is a schematic diagram of a landscape mode of a mobile terminal of the present application;

3B is a schematic diagram of a portrait mode of the mobile terminal of the present application;

4 is a schematic diagram of receiving a horizontal media resource when a display device is in a landscape orientation according to an embodiment of the application;

FIG. 5 is a schematic diagram of receiving a horizontal media resource and rotating to a horizontal screen when the display device is in a vertical screen according to an embodiment of the present application;

6 is a schematic diagram of receiving a vertical media resource when a display device is in a vertical screen according to an embodiment of the present application;

FIG. 7 is a schematic diagram of receiving a vertical media resource and rotating to a vertical screen when the display device is in a landscape orientation according to an embodiment of the present application;

8 is a schematic flowchart of a processing method for receiving screen projection data according to the present application;

FIG. 9 is a schematic flow chart of the application comparing sampling resolution and initial resolution;

10A is a schematic diagram of a mobile terminal in a landscape mode sending image information to a display device in a landscape state according to an embodiment of the application;

10B is a schematic diagram of a mobile terminal in a landscape mode sending image information to a display device in a portrait screen state according to an embodiment of the application;

11A is a schematic diagram of a mobile terminal in a portrait mode sending image information to a display device in a landscape state according to an embodiment of the application;

11B is a schematic diagram of the effect of presenting a picture when the display device is in a vertical screen state according to the screen projection protocol;

11C is a schematic diagram of a mobile terminal in a portrait mode sending image information to a display device in a portrait state according to an embodiment of the application;

12 is a schematic flowchart of a method for detecting an effective resolution of a projected video stream according to the present application;

13 is a schematic flow chart of the application for comparing sampling resolution and initial resolution;

FIG. 14A is a schematic flowchart of calculating a reference value and a comparison value for the application;

14B is a schematic diagram of black border data and initial screen data of the present application;

15 is a schematic flowchart of the application for judging preset execution conditions;

16 is a schematic flowchart of the application comparing multiple sampling resolutions and initial resolutions;

FIG. 17 is a schematic flowchart of the execution of a controller of an exemplary display device of the present application;

18 is a schematic flowchart of the application for determining the magnification of effective information according to the ratio of the height and width of the display device and the effective screen;

19 is a schematic structural diagram of a display device of the application;

20A is a schematic diagram of a mobile terminal in a portrait mode sending image information to a display device in a portrait state according to an embodiment of the application;

20B is a schematic diagram of a display device when the mobile terminal is in a portrait mode and sends image information to a portrait screen state and the picture is enlarged by 1 times in accordance with an embodiment of the application;

20C is a schematic diagram of a display device when the mobile terminal is in a portrait mode and sends image information to a portrait screen state and magnifies the picture by 3 times according to an embodiment of the present application.

Detailed ways

Embodiments will be described in detail below, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following examples are not intended to represent all implementations consistent with this application. are merely exemplary of systems and methods consistent with some aspects of the present application as recited in the claims.

In the technical solution provided by the present application, the display device may be an electrical device with a larger screen, such as a smart TV, that presents video and audio signals to the user. The display device can have an independent operating system and support function expansion. Various applications can be installed in the display device according to user needs, for example, traditional video applications, social applications such as short videos, and reading applications such as comics and book reading. These applications can use the screen of the display device to display the application screen and provide users with richer media resources. At the same time, the display device can also perform data interaction and resource sharing with different terminals. For example, a smart TV can be connected to a mobile phone through a wireless communication method such as a local area network or Bluetooth, so as to play the resources in the mobile phone or directly perform screen projection to display the picture on the mobile phone.

Embodiments of the present application provide a display device and a computer storage medium. The display device, such as a TV that can support horizontal and vertical screen rotation, is referred to as a rotatable TV. It should be noted that the method provided in this embodiment is not only applicable to rotatable TVs, but also applicable to other display devices, such as computers, tablet computers, etc., so as to facilitate the user to display the details page of the target media asset in different horizontal and vertical screen states of the display, so as to improve the Displays the user viewing experience of the device in different viewing states.

The term "module" used in the various embodiments of the present application may refer to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware or/and software codes capable of executing the execution associated with the element function.

The term "remote control" used in various embodiments of the present application refers to a component of an electronic device (such as the display device disclosed in the present application), which can usually wirelessly control the electronic device within a short distance range. The component can generally use infrared and/or radio frequency (RF) signals and/or Bluetooth to connect with electronic devices, and may also include functional modules such as WiFi, wireless USB, Bluetooth, and motion sensors. For example, a hand-held touch remote control replaces most of the physical built-in hard keys in a general remote control device with a user interface in a touch screen.

The term "gesture" used in various embodiments of the present application refers to a user behavior that is used by a user to express an expected idea, action, purpose/or result through an action such as a change of hand shape or hand movement.

The term "hardware system" used in the various embodiments of this application may refer to a computer system composed of mechanical, optical, electrical, and magnetic devices such as an integrated circuit (IC) and a printed circuit board (PCB). , physical components that control, store, input, and output functions. In various embodiments of the present application, the hardware system is also commonly referred to as a motherboard or a main chip or a controller.

Referring to FIG. 1 , it is an application scenario diagram of a display device provided by some embodiments of the present application. As shown in FIG. 1 , communication between the control apparatus 100 and the display device 200 may be performed in a wired or wireless manner.

The control device 100 is configured to control the display device 200 , which can receive operation instructions input by the user, and convert the operation instructions into instructions that the display device 200 can recognize and respond to, acting as an intermediary for the interaction between the user and the display device 200 . effect. For example, the user operates the channel addition and subtraction keys on the control device 100, and the display device 200 responds to the channel addition and subtraction operation.

The control apparatus 100 may be a remote controller 100A, including infrared protocol communication or Bluetooth protocol communication, and other short-distance communication methods, etc., and controls the display device 200 by wireless or other wired methods. The user can control the display device 200 by inputting user instructions through keys on the remote control, voice input, control panel input, and the like. For example, the user can control the display device 200 by inputting corresponding control commands through the volume up/down key, channel control key, up/down/left/right movement keys, voice input key, menu key, power-on/off key, etc. on the remote control. function.

The control apparatus 100 may also be a smart device, such as a mobile terminal 100B, a tablet computer, a computer, a notebook computer, and the like. For example, the display device 200 is controlled using an application running on the smart device. The app can be configured to provide users with various controls through an intuitive user interface (UI) on the screen associated with the smart device.

Exemplarily, the mobile terminal 100B can install a software application with the display device 200, and realize connection communication through a network communication protocol, so as to realize the purpose of one-to-one control operation and data communication. For example, the mobile terminal 100B and the display device 200 can be made to establish a control instruction protocol, and by operating various function keys or virtual controls of the user interface provided on the mobile terminal 100B, the functions of the physical keys arranged by the remote control 100A can be realized. The audio and video content displayed on the mobile terminal 100B may also be transmitted to the display device 200 to implement a synchronous display function.

The display device 200 may be a liquid crystal display, an organic light emitting display, or a projection device. The specific display device type, size and resolution are not limited.

The display device 200 also performs data communication with the server 300 through various communication methods. Here, the display device 200 may be allowed to be communicatively connected through a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 300 may provide various contents and interactions to the display device 200 . By way of example, display device 200 may send and receive information, such as receiving electronic program guide (EPG) data, receiving software program updates, or accessing a remotely stored digital media library. The server 300 may be in one group, or in multiple groups, or in one or more types of servers. Other network service contents such as video-on-demand and advertising services are provided through the server 300 .

A block diagram of a hardware configuration of the display device 200 is exemplarily shown in FIG. 2 . As shown in FIG. 2, the display device 200 may include a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a memory 260, a user interface 265, a video processor 270, a display 275, Rotating assembly 276 , audio processor 280 , audio output interface 285 , power supply 290 .

The rotating assembly 276 may include components such as a drive motor, a rotating shaft, and the like. The drive motor can be connected to the controller 250, and is controlled by the controller 250 to output a rotation angle; one end of the rotating shaft is connected to the power output shaft of the drive motor, and the other end is connected to the display 275, so that the display 275 can be fixedly installed on the rotating assembly 276. on a wall or stand.

The rotating assembly 276 may also include other components, such as transmission components, detection components, and the like. The transmission component can adjust the rotational speed and torque output by the rotating component 276 through a specific transmission ratio, which can be a gear transmission mode; the detection component can be composed of sensors arranged on the rotating shaft, such as an angle sensor, an attitude sensor, and the like. These sensors can detect parameters such as the rotation angle of the rotating component 276, and send the detected parameters to the controller 250, so that the controller 250 can judge or adjust the state of the display device 200 according to the detected parameters. In practical applications, the rotating assembly 276 may include, but is not limited to, one or more of the above components.

The tuner and demodulator 210 receives broadcast television signals through wired or wireless means, and can perform modulation and demodulation processing such as amplification, frequency mixing, and resonance, so as to demodulate the television selected by the user from multiple wireless or wired broadcast television signals The audio and video signals carried in the frequency of the channel, as well as additional information (eg EPG data). In some other exemplary embodiments, the tuner 210 may also be in an external device, such as an external set-top box or the like. In this way, the set-top box outputs the TV signal after modulation and demodulation, and then inputs the TV signal to the display device 200 through the external device interface 240 .

The communicator 220 is a component for communicating with external devices or external servers according to various communication protocol types. For example, the display device 200 may transmit content data to an external device connected via the communicator 220 , or browse and download content data from an external device connected via the communicator 220 . The communicator 220 may include a network communication protocol module such as a WIFI module 221 , a Bluetooth communication protocol module 222 , a wired Ethernet communication protocol module 223 or a near field communication protocol module, so that the communicator 220 can receive the information from the control device 100 according to the control of the controller 250 . Control signal, and realize the control signal as WIFI signal, Bluetooth signal, radio frequency signal, etc.

The detector 230 is a component of the display device 200 for collecting external environment or external interaction signals. The detector 230 can include a sound collector 231, such as a microphone, which can be used to receive the user's voice, such as the voice signal of the user's control instruction to control the display device 200; Device 200 can adapt to ambient noise.

The external device interface 240 is a component that provides the controller 250 to control data transmission between the display device 200 and the external device. The external device interface 240 can be connected with external devices such as set-top boxes, game devices, notebook computers, etc. in a wired/wireless manner, and can receive external devices such as video signals (such as moving images), audio signals (such as music), additional information (such as EPG) and other data.

The controller 250 controls the operation of the display device 200 and responds to user operations by running various software control programs (such as an operating system and various application programs) stored in the memory 260 .

In some exemplary embodiments, controller 250 includes random access memory (RAM), read only memory (ROM), a graphics processor, a CPU processor, a communication interface, and a communication bus. The RAM, the ROM, the graphics processor, and the CPU processor communication interface are connected through a communication bus. ROM, used to store various system startup instructions. For example, when the power-on signal is received, the power supply of the display device 200 starts, and the CPU processor 254 executes the system startup instruction in the ROM 252, and copies the operating system stored in the memory 260 to the RAM 251 to start the operating system. After the startup of the operating system is completed, the CPU processor 254 copies the various application programs in the memory 260 to the RAM 251, and then starts to run and activate the various application programs.

The communication interface 255 may include a first interface to an nth interface. These interfaces may be network interfaces connected to external devices via a network.

The controller 250 may control the overall operation of the display apparatus 200 . For example, in response to receiving a user input command for selecting a GUI object displayed on the display 275, the controller 250 may perform operations related to the object selected by the user input command.

where the object can be any of the optional objects, such as a hyperlink or an icon. The operation related to the selected object, for example, the operation of displaying links to hyperlinked pages, documents, images, etc., or the operation of executing the program corresponding to the object. The user input command for selecting a GUI object may be an input command through various input devices (eg, mouse, keyboard, touchpad, etc.) connected to the display device 200 or a voice command corresponding to the voice spoken by the user.

The memory 260 is used to store various types of data, software programs or application programs that drive and control the operation of the display device 200 . Memory 260 may include volatile and/or nonvolatile memory. Whereas, the term “memory” includes the memory 260 , the RAM and ROM of the controller 250 , or a memory card in the display device 200 .

In some embodiments, the memory 260 is specifically used to store the operating program for driving the controller 250 in the display device 200 ; to store various application programs built in the display device 200 and downloaded by the user from external devices; Various GUIs, various GUI-related objects, and visual effect images of selectors used to select GUI objects.

In some embodiments, memory 260 is specifically used to store drivers and related data for tuner demodulator 210, communicator 220, detector 230, external device interface 240, video processor 270, display 275, audio processor 280, etc. , such as external data (such as audio and video data) received from an external device interface or user data (such as key information, voice information, touch information, etc.) received from a user interface.

In Figure 2, user interface 265, receives various user interactions. Specifically, it is used to send the user's input signal to the controller 250, or to transmit the output signal from the controller 250 to the user. Exemplarily, the remote control 100A may send input signals input by the user, such as power switch signals, channel selection signals, volume adjustment signals, etc., to the user interface 265, and then the user interface 265 forwards them to the controller 250; or, the remote control 100A may The controller 250 processes an output signal such as audio, video or data output from the user interface 265, and displays the received output signal or outputs the received output signal in the form of audio or vibration.

In some embodiments, the user may input user commands on a graphical user interface (GUI) displayed on the display 275, and the user interface 265 receives the user input commands through the GUI. Specifically, user interface 265 may receive user input commands for controlling the position of selectors in the GUI to select different objects or items. Among them, the "user interface" is the medium interface for interaction and information exchange between the application program or the operating system and the user, which realizes the conversion between the internal form of the information and the form acceptable to the user. A commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed graphically. It can be an icon, window, control and other interface elements displayed on the display screen of the electronic device, wherein the control can include icons, controls, menus, tabs, text boxes, dialog boxes, status bars, channel bars, Widgets, etc. visual interface elements.

Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user interface 265 recognizes the sound or gesture through a sensor to receive the user input command.

The video processor 270 is used to receive an external video signal, and perform video data processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to the standard codec protocol of the input signal, and can obtain A video signal that is displayed or played directly on display 275 .

The display 275 is used for receiving the image signal input from the video processor 270 to display the video content, the image and the menu manipulation interface. The displayed video content can be from the video content in the broadcast signal received by the tuner and demodulator 210 , or from the video content input from the communicator 220 or the external device interface 240 . The display 275 simultaneously displays a user manipulation interface UI generated in the display device 200 and used to control the display device 200 .

And, the display 275 may include a display screen component for presenting a picture and a driving component for driving image display. Alternatively, if the display 275 is a projection display, it may also include a projection device and a projection screen.

Rotating assembly 276, the controller may issue a control signal to cause rotating assembly 276 to rotate display 255.

The audio processor 280 is used for receiving an external audio signal, decompressing and decoding according to the standard codec protocol of the input signal, and performing audio data processing such as noise reduction, digital-to-analog conversion, and amplification processing to obtain a signal that can be displayed in the speaker 286. Played audio signal.

The power supply 290 is used to provide power supply support for the display device 200 with the power input from the external power supply under the control of the controller 250 . The power supply 290 may be a built-in power supply circuit installed inside the display device 200 , or may be a power supply installed outside the display device 200 .

During the screen projection process, the mobile terminal 100B may send display screen data to the display device 200 through a wireless connection, such as through the Miracast protocol or the DLNA protocol, to form a screen projection video stream. After receiving the screen projection video stream, the display device 200 can decode the screen projection video stream through the controller 250, parse out the frame images and process them to form the screen projection images and send them to the display 275 for display.

The following description will be given by taking the mobile terminal projecting a screen to a TV as an example. The mobile terminal 100B may be an intelligent terminal device with display and human-computer interaction functions, such as a mobile phone, a tablet computer, and the like. The video or picture on the mobile terminal may be content whose horizontal orientation is in the positive orientation, such as a TV drama played when the mobile phone is placed horizontally. When the phone is placed in landscape orientation, the aspect ratio of the screen is greater than 1. The video or picture on the mobile terminal may also be the content whose vertical mode is the positive direction, such as a picture taken when the user's mobile phone is placed vertically. When the phone is placed in portrait orientation, the aspect ratio of the screen is less than 1.

In some embodiments, the TV can support horizontal and vertical rotation driven by the rotating assembly. For example, when a TV series or a movie is played, since these media resources are generally horizontal resources, the TV is placed horizontally to present horizontal media resources. When playing fitness or vertical media resources, the TV is placed vertically to present vertical resources.

In some embodiments, the TV is placed horizontally, and when the rotation instruction is not triggered, it can also present the received screen projection and present the vertical resource. And the TV is placed vertically. When the rotation command is triggered, it can also present the received screen projection and present the horizontal resource.

In some embodiments, when screen sharing is performed through the DLNA protocol, the display device can obtain the horizontal and vertical presentation states of the image. Since the mobile terminal 100B can present media resources in a horizontal or vertical layout, the projected screen also has different layout modes. . For example, when a user presents a horizontal media resource on a mobile phone, the width of the screen is greater than the height of the screen, as shown in FIG. 3A . When the user presents vertical media resources on the mobile phone, the width of the screen is smaller than the height of the screen, as shown in FIG. 3B .

In order to enable the display device 200 to automatically rotate the screen according to the horizontal and vertical states of the screen of the mobile terminal 100B during screen projection, so as to achieve a better user experience, a screen projection protocol may be configured between the display device 200 and the mobile terminal 100B, for example, based on DLNA A standard screen projection protocol, a screen projection protocol based on the Airplay standard, etc., the transmission of the picture and video stream is realized through the screen projection protocol, so that the picture on the mobile terminal 100B can be projected to the display device 200 .

In some embodiments, the projected content may be a video or a picture. In general, users browse videos for a long time, while browsing pictures is relatively short.

When the projected screen content received by the display device is a video, the horizontal and vertical screen relationship of the video content can be detected, and combined with the current display presentation mode to determine whether to rotate. Then keep the original state and play the video directly, as shown in Figure 4. If the horizontal video is received in the vertical screen state, and it is intelligently presented on part of the display interface due to the inconsistent aspect ratio, then while the video is played in the vertical screen first, the rotation component is controlled to rotate the display device 275 to the horizontal screen state. Full-screen playback has been achieved, as shown in Figure 5.

If the vertical video is received in the vertical screen state, full-screen playback can be achieved, and the video can be played directly in the original state, as shown in Figure 6. If a vertical video is received in the landscape state, and it is intelligently presented on part of the display interface due to inconsistent aspect ratios, then the rotation component is controlled to rotate the display 275 of the display device to the portrait state while the video is first played in the landscape state. , full-screen playback has been achieved, as shown in Figure 7.

When the projected content is a picture, the user may be browsing the application related to the photo collection. At this time, the user can stay on a single image for a limited time. One picture becomes vertical, and the next one becomes horizontal. Moreover, when the presentation state of the display device changes, that is, it takes a certain amount of time for the rotating component to drive the display device to rotate, and the display device placed in the horizontal screen just receives the vertical resource and starts to rotate to the vertical, the user may trigger the next page to browse. The instructions need to be changed back to landscape again. Such frequent rotation of the display device not only affects the user experience, but also reduces the service life of the display device.

In some embodiments, in a display device of the present application, it is necessary to determine whether the display needs to be rotated according to the received media resource information, the horizontal and vertical states of the media resources, and the presentation state of the display itself. For video playback, when the mobile phone terminal pushes, the TV terminal rotates and zooms in/out of the TV and video playback interface with the video screen, always maintains the vertical state of the video, and finally adapts to the display effect of the entire screen. For picture playback, the horizontal and vertical screens of the TV end shall prevail. The horizontal screen means that the horizontal screen is displayed at the maximum scale according to the original scale, and the vertical screen means that the vertical screen is displayed at the maximum scale according to the original scale. The rotation angle is reset when rotating, and finally the display screen is adapted to the maximum scale.

In some embodiments, for video playback: when the TV terminal receives the video link from the mobile terminal, it accesses the video resources, determines the horizontal and vertical screens of the TV terminal, and registers the angle monitoring, and then adapts the entire video terminal according to the original proportion after obtaining the width and height of the video. The screen is displayed and played. When the width and height of the video are inconsistent with the horizontal and vertical screens of the TV, a rotation command is sent, and the angle information of the current TV rotation is obtained through the angle monitoring. The zoom factor is set to the frame capture interface. When the frame capture interface obtains the angle information and zoom factor, the angle and zoom settings of the current video playback screen are set, and the image is always displayed vertically until the rotation ends.

In some embodiments, for picture playback: when the TV terminal receives a picture push request from the mobile phone terminal, it determines the horizontal and vertical screen of the current TV terminal, and displays the entire screen with the original ratio of the current image width and height. When the user's remote control manually rotates the TV When listening to ConfigurationChange, set the angle and set the zoom factor to ensure that the picture screen is displayed on the entire screen according to the original ratio.

A display device provided by this application includes:

monitor;

a rotation component, configured to drive the display to rotate, so that the display is in one of a horizontal screen state or a vertical screen state;

a user interface, the communication is configured to connect to a terminal;

And the controller, see Figure 8, is configured to execute:

S01: Receive screen projection media resources, and determine attributes of the media resources;

In practical applications, the user may first perform a screen projection display operation on the mobile terminal 100B to send the display screen of the mobile terminal 100B to the display device 200 . For example, the user selects "settings - connection and sharing - screencasting" on the mobile phone, and selects a display device in the current network as the screencasting object in the screencasting device list to perform the screencasting operation.

After performing the screen projection operation, the mobile terminal 100B will send the displayed picture to the display device 200 through a screen projection protocol, such as using the DLNA protocol or other mirroring protocols. As new interactive pictures are continuously generated during the screen projection process, the mobile terminal 100B will send the pictures to the display device 200 frame by frame to form a screen projection data stream (hereinafter referred to as a screen projection video stream).

It should be noted that users can also perform screen casting operations through third-party applications. For example, when a user opens a video application, a screen-casting icon is provided on the video playback interface of the video application. Users can click this icon to perform screen casting operations. Generally, the screencasting of the screencasting operation performed by a third-party application is based on the played video resource. For example, when the video resources played are horizontal media assets such as movies and TV series, the width of the effective screen in the projection screen is greater than the height; when the video resources played are vertical media resources such as short videos and comics, the screen projection screen The width of the effective picture in is less than the height.

The TV receives media resources, and judges whether it is a video or an image according to the attributes of the media resources.

S02: If the media resource attribute is video, detect the horizontal and vertical screen states of the display, obtain the video aspect ratio, and determine the matching situation between the video aspect ratio and the horizontal and vertical screen states of the display, and determine the The match determines whether the display needs to be rotated.

After receiving the screencast video stream, the display device 200 starts to register the monitoring of the rotation angle, recalls the current real-time rotation angle data of the machine, and obtains the current rotation direction and angle information of the TV, and then judges the horizontal and vertical screen status of the TV. And obtain the width and height data of the video, first display the video screen in the original ratio on the TV side, and compare the horizontal and vertical screen of the TV side with the video width and height. , the video is horizontal screen, it will drive the TV to rotate automatically.

S03: If the media resource attribute is a picture, fill the screen according to the original aspect ratio of the current image for display.

In an implementation manner, in order to realize that the video aspect ratio does not match the horizontal and vertical screen states of the display, while driving the display to rotate, the video may be synchronously enlarged. Obtain the current rotation angle in real time, calculate the zoom factor, rotate the UI in the opposite direction at the same time through the rotation direction and angle data, and simultaneously zoom in or zoom out the display effect.

In some embodiments, driving the display to rotate when the video aspect ratio does not match the horizontal and vertical screen states of the display. When the aspect ratio of the video matches the horizontal and vertical screen states of the display, it is directly displayed on the display to adapt the screen according to the original ratio to present the video content.

In some exemplary embodiments, not all videos can trigger rotation, and it is also necessary to determine whether to rotate according to the video duration, so that when the projected content is a longer video, the video aspect ratio and the display state can match the display state. The user experience is better. When the video time is too short, the user may play the last shorter video, and the display device will end the display as soon as the display rotates, and wait for the next video projected by the user, and then according to the aspect ratio Again, rotating the display so frequently will give the user a bad user experience.

For example, the controller can start playing on the user interface after receiving the projected screen content, and then detect the video aspect ratio and the presentation state of the display when the video duration is greater than or equal to a certain time value T _min . When the aspect ratio does not match the presentation state of the display, the rotating component is controlled to drive the display to rotate. When the video duration is less than the time value T _min , it is determined that the video duration is too short, and the TV does not need to be rotated.

As shown in FIG. 9 , if the aspect ratio of the video does not match the horizontal and vertical screen states of the display, driving the display to rotate, further comprising:

S021: Obtain the original resolution of the video that is adapted to the display before rotation;

S022: Determine the zoom factor of the video picture according to the original resolution and the full-screen resolution of the display;

S023: Monitor the rotation angle information of the display, and gradually scale the video image according to the rotation angle information to ensure that the video image is finally displayed in full screen.

In some embodiments, the initial resolution is F1 and the full-screen resolution is F2, and the video picture needs to be enlarged by F2/F1 times during the rotation process. After listening to the rotation angle of the display from landscape to portrait, the entire rotation process can be divided into 90 parts according to the rotation angle, and each degree of rotation will be enlarged (F2/F1)/90. This rotates to full screen at a constant speed.

For example, when the display device is in the horizontal screen state and receives vertical video resources, the user interface presented by the display is shown in (a) in Figure 7: the length of the vertical side of the video image is the same as the length of the long side of the display, and the display . Since the display device is in the horizontal screen state and the video interface is in the vertical screen state at this time, the horizontal and vertical screens of the TV terminal and the video screen do not match, so the rotation interface of the display device is called to control the rotating component to drive the rotating display. The controller obtains the rotation direction and angle information through the rotation angle callback, and the change of the video playback screen is calculated in real time according to the angle information of the TV rotation. The angle data and zoom factor are set to the frame capture interface, and the video screen is rotated and zoomed in real time. The video playback screen is always in a vertical state relative to the user. At this time, the TV screen and the video screen are overlapped and displayed, as shown in Figure 7 (c ).

In some embodiments, the magnification factor can also be calculated according to the length of the long side of the display and the height of the video picture, and the length of the short side of the display and the width of the video picture. For example, the default multiplier is 1 when the rotation starts, and the video playback screen is enlarged by 1 degree in height (long side length of TV screen - video screen height)/90, and wide zoom (short side width of TV screen - video screen width)/90.

In some embodiments, before the rotation process starts, a relevant prompt or rotation may be popped up on the user interface, and then the display is driven to rotate.

In some embodiments, when the display device is displayed in a vertical screen and the video resource is in a horizontal state, the user interface presented by the display is shown in (b) in FIG. 5 : the length of the horizontal side of the video image is the same as the length of the short side of the display, and which shows. At this time, the horizontal and vertical screens of the display device and the video screen do not match, so the rotation interface of the TV terminal is called to notify the TV terminal to rotate, and the rotation direction and angle information are obtained through the rotation angle callback, and the change of the video playback screen is calculated in real time according to the angle information of the TV rotation. As an optional method, when the TV end rotates from landscape to portrait, the default multiplier at the beginning of the rotation is 1, and when the rotation is 1 degree, the video playback screen is highly enlarged (the length of the short side of the TV screen - the height of the video screen)/90, Width zoom (long side length of TV screen - width of video screen)/90, set the current angle data and zoom factor to the frame capture interface, rotate and zoom the video screen in real time, and the video playback screen is always in a vertical state relative to the user , and the final TV screen and the video playback screen are displayed superimposed, as shown in (c) in FIG. 5 .

In some embodiments, after the display device receives a picture push request from the mobile phone, the display device is displayed at the maximum original scale of the current picture, regardless of whether the display of the picture and the display match whether the display is rotated.

Of course, the above-mentioned screen projection of pictures is only an exemplary illustration. In some embodiments, regardless of whether the projected media asset is a video or a picture, when it does not match the display state of the display, the rotation component can be triggered to perform the display operation on the display. rotate.

In some embodiments, the display device is not only rotated according to the screen projection data, no matter whether the display device is in a landscape state or a portrait state, the user can still actively issue a rotation instruction to rotate the display device to rotate the TV to a different level than before. the state of. Viewed from the front of the display device, the rotating component 276 can rotate the display screen to a vertical screen state, that is, a state in which the vertical side length of the screen is greater than the horizontal side length, or can rotate the screen to a landscape screen state, that is, the screen is in a horizontal screen state. A state where the side length is greater than the vertical side length. For example, the user can manually trigger the display to rotate, and when a subjectively conscious instruction is input, the priority execution level can be set to be high. For example, a rotation instruction input by the user is received regardless of whether the presentation of the video picture and the display match, even if the video picture and the presentation of the display match, a trigger instruction may be received to rotate the display.

In some embodiments, the screencasting protocol is the Miracast screencasting protocol.

According to the current Miracast screen projection protocol, regardless of whether the mobile terminal 100B is in landscape mode or portrait mode, the video stream projected to the display device 200 is always a 1920×1080 landscape stream. The aspect ratio obtains the horizontal and vertical relationship of the screen, so that the display device 200 cannot realize the automatic rotation of the TV screen through the video stream.

For example, when a mobile terminal interacts with a display device for screen projection, when the mobile terminal is in a landscape mode, no matter whether the display device interacting with the mobile terminal is in a landscape state or a portrait state, the image information sent to the display device will not be changed. Contains left and right black information, as shown in Figures 10A and 10B.

When the mobile terminal is in portrait mode, no matter whether the display device interacting with the mobile terminal is in landscape or portrait state, the image information sent to the display device includes left and right black information, as shown in FIGS. 11A and 11B .

In the Miracast protocol, when the mobile terminal establishes a communication connection with the display device, before the mobile terminal transmits the screenshot to the display device, the screenshot has been processed by adding black borders to ensure that the original display device is in the horizontal screen state. In this case, regardless of whether the mobile terminal is placed horizontally or vertically, a horizontal screen can be displayed.

In the projection screen corresponding to FIG. 11A , the image information shown in the black areas on both sides of the screen is black information, or called black borders, and the displayed screen area in the middle of the screen is called the effective screen (that is, the image corresponding to the mobile terminal). screen display picture), the effective picture is the effective information in the image information of the mobile terminal. In the area corresponding to the effective screen, the operation screen on the mobile terminal 100B is displayed, and the black border has different widths and heights according to the screen ratio of the display 275 of the mobile terminal 100B.

That is, when the display device 200 displays the screen projection image, the direction corresponding to the shorter side is used as a reference, such as the height direction in the horizontal screen state. Therefore, the height direction of the projected video stream presented by the display device 200 is generally unchanged. That is, regardless of whether the placement direction of the mobile terminal 100B is horizontal or vertical, the height of the projected screen received by the display device 200 is 1080P. Even if the display 275 is rotated to the vertical screen state, it cannot display the display state shown in FIG. 11C , but is displayed according to the height of the entire image information including left and right black information, as shown in FIG. 11B . That is, in the vertical screen state, not only the left and right sides of the effective screen are filled with black information, but the top and bottom of the projected screen also display black areas because there is no effective screen to fill, which will greatly affect the user's viewing experience.

In the actual screen projection process, the screen projection video stream received by the display device 200 corresponds to a screen display resolution of 1920×1080, that is, in the height direction, the screen projection image is based on the screen on the terminal, and requires 1080 pixels for display. . In the width direction, the projection screen is based on the terminal display screen plus the width of the black borders on both sides, and requires 1920 pixels to display. Correspondingly, the height of the effective area in the projection screen is 1080, and the width is scaled and displayed in proportion to the height, which will greatly waste the display area on the display 275 .

In order to improve the utilization rate of the display area on the display device, the effective information is enlarged in a certain proportion to adapt to the screen projection video stream sent by the mobile terminal 100B.

In some embodiments, when the mobile terminal is in landscape mode, the state shown in FIG. 3A is presented, and when screencasting is performed, the parsed frame data of the video stream sent to the display device has no left and right black borders. When the display device receives image information that does not contain left and right black borders, it is detected whether the display is in a landscape state. When the screen is in a landscape state, the controller 250 receives the image information, and can directly present the image information on the display. When the display is in the portrait state, the controller 250 may send a rotation instruction to the rotation component 276 to control the rotation component 276 to rotate, so as to rotate the display to the landscape state.

When the mobile terminal is in portrait mode, the state shown in FIG. 3B is presented, and when screen projection is performed, the parsed frame data of the video stream sent to the display device has left and right black borders and an effective picture. When the display device receives image information with left and right black borders, it detects whether the display is in a vertical screen state. When in the landscape state, the controller 250 may send a rotation instruction to the rotation component 276 to control the rotation component 276 to rotate the display 275 to the portrait state. In the vertical screen state, the width of the display 275 is smaller than the height, which is in line with the display ratio of the terminal screen. At this time, the frame data in the projected video stream is enlarged, and the effective screen is displayed on the display. The left and right black areas are enlarged due to the enlargement. cannot be displayed on the display, thereby reducing the black border area on both sides of the effective screen, as shown in Figure 11C.

In order to improve the user's viewing experience and reduce the black information on both sides of the screen, the present application provides a display device, which can judge whether the display needs to be rotated and the screen needs to be rotated according to whether the image information of the projected screen data stream contains left and right black information. Make adjustments to maximize the use of the display area.

A display device provided by this application includes:

monitor;

a user interface, the communication is configured to connect to a terminal;

And the controller, see Figure 12, is configured to execute:

S11: Receive a screencast video stream.

After performing the screen projection operation, the mobile terminal 100B will send the displayed picture to the display device 200 through a screen projection protocol, such as using the Miracast protocol or other mirroring protocols. As new interactive pictures are continuously generated during the screen projection process, the mobile terminal 100B will send the pictures to the display device 200 frame by frame to form a screen projection data stream (hereinafter referred to as a screen projection video stream).

S12: Extract the initial resolution from the screen projection video stream.

After the display device 200 receives the screen projection video stream, the controller 250 of the display device 200 may perform frame-by-frame analysis on the received screen projection video stream to extract the initial resolution. The initial resolution is the overall resolution of the first frame in the projected video stream.

For example, the aspect ratio of the screencast video stream sent by the mobile terminal 100B is 1920:1080. After receiving the screencast video stream, the controller 250 can obtain each frame of the picture by parsing the screencast video stream. And the initial resolution is extracted from the first frame picture (the first frame picture), that is, the extracted initial resolution is 1920×1080.

Obviously, the first frame picture is not limited to the first frame picture in the whole screen projection process, but may also be the first few frames of the screen projection video stream or the corresponding frame pictures at a specified time node. Since the initial resolution is the overall resolution of the projected screen, it is easier to obtain the resolution information, and in practical applications, the earlier the initial resolution is obtained, the more conducive to timely detection of the effective resolution of the projected video stream. Therefore, the display screen can be adjusted to a better state as soon as possible.

S13: Extract the sampling resolution from the screen projection video stream.

After the initial resolution is extracted, the images in the screen projection video stream can be further sampled to extract the sampling resolution. The frame of picture used for sampling is called a sampling picture. The sampling resolution is the resolution of the effective area (ie, the effective picture) on the sampling picture, which is extracted from the projection video stream according to a preset time interval.

During the sampling process, the preset time interval for the designated sampling may be set according to the computing capability of the controller 250 , for example, a picture 10 s after the first frame picture is taken as a sampling picture. In order to determine an effective picture from the sampled picture, the pixel color of the sampled picture can be traversed. Obviously, the pixel color value of the black border area is black, and the pixel color value of the effective area is usually not all black. Therefore, it can be determined by traversing each pixel of the sampled image that the black and rectangular area is the black border, and the other areas are valid area.

S14: Compare the sampling resolution with the initial resolution.

After extracting the initial resolution and sampling resolution for the projected video stream, the sampling resolution can be compared with the initial resolution, so as to determine the valid data in the current projected video stream according to the difference between the sampling resolution and the initial resolution. Screen conditions (such as scale, orientation, etc.), so as to choose whether to display according to the effective screen.

For example, if the sampling resolution is equal to the initial resolution, that is, the resolution of the first frame is 1920×1080, and the resolution of the effective area determined in the sampling image is also 1920×1080, it means that there is no blackness in the current projected screen. On the other hand, the projected screen can fill the display area. That is, the display requirements of the screen projection image can be satisfied by directly displaying the screen in the horizontal screen state of the display 275 .

It should be noted that, because the resolution of the display screen is usually represented by the number of pixels occupied in the width and height directions of the screen, for example, 1920×1080. However, it is usually difficult to directly compare the value of resolution alone. For example, from a numerical comparison, a resolution of 1920×1080 is equal to 1080×1920. Therefore, in the actual comparison process, a comparison result between the sampling resolution and the initial resolution can be obtained by extracting part of the value in the resolution or converting the resolution into other comparable values, and then performing the comparison. . For example, the width or height of the overall picture can be extracted from the initial resolution, and compared with the height or width of the effective picture extracted from the sampling resolution to determine its effective resolution.

S15: If the sampling resolution is greater than or equal to the initial resolution, set the effective resolution of the video stream as the initial resolution.

In practical applications, if the value corresponding to the sampling resolution is greater than or equal to the value corresponding to the initial resolution, it means that the black border area in the current projection screen is the smallest on the premise of ensuring complete display. At this time, even if the screen is zoomed, the area of the small black border area will not be increased, so it is determined that the effective resolution of the video stream has not changed and is still the initial resolution.

S16: If the sampling resolution is smaller than the initial resolution, set the effective resolution of the video stream as the sampling resolution.

If the value corresponding to the sampling resolution is smaller than the value corresponding to the initial resolution, it is determined that the current projection image fills a large amount of black areas, and the display device 200 can improve the display effect by rotating the display 275 and scaling the projection image.

It can be seen from the above technical solutions that the method for detecting the effective resolution of a screencast video stream provided by the present application can extract the initial resolution and the sampling resolution from the screencast video stream after receiving the screencast video stream, and compare them to determine the current video. The effective resolution of the stream. If the sampling resolution is smaller than the initial resolution, set the effective resolution of the video stream to the sampling resolution. By setting the effective resolution of the video stream, the projected screen can be displayed according to the effective resolution, so as to adapt to the display direction of the projected screen, reduce the influence of black borders, and achieve a better user experience.

In an implementation manner, in order to achieve the sampling resolution and the initial resolution, data that can be used for comparison may be extracted from the sampling resolution and the initial resolution, respectively. That is, as shown in FIG. 13 , the step of comparing the sampling resolution and the initial resolution further includes:

S141: Extract a reference value according to the initial resolution;

S142: Traverse the valid pictures of the sampling pictures to generate a contrast value;

S143: If the contrast value is greater than or equal to the reference value, determine that the sampling resolution is greater than or equal to the initial resolution;

S144: If the contrast value is smaller than the reference value, determine that the sampling resolution is smaller than the initial resolution.

After the initial resolution is obtained, some data can be extracted from the initial resolution as a reference value. For example, when the height of the projected image in the projected video stream does not change, the overall image height of the first frame image is used as the reference value. Then, by traversing the pixel color values in the sampled picture, the ratio of the effective picture is determined, thereby generating the contrast value. The contrast value is a picture width of an effective picture in the sampled picture.

For example, if the initial resolution extracted from the first frame is 1920×1080, the extraction reference value is 1080. Then, by traversing the pixel color values in the sampled image, the resolution of the effective image after removing the black area is determined as

960×1080, resulting in a contrast value of 960.

After the reference value and the contrast value are determined, the relationship between the sampling resolution and the initial resolution can be determined directly by comparing the size of the reference value and the contrast value, that is, if the contrast value is greater than or equal to the reference value, it is determined that the sampling resolution is greater than or equal to Equal to the initial resolution; if the contrast value is smaller than the reference value, it is determined that the sampling resolution is smaller than the initial resolution.

For example, the reference value is 1080 and the contrast value is 960. Since the contrast value of 960 is smaller than the reference value of 1080, it is determined that the sampling resolution is smaller than the initial resolution, that is, the effective resolution of the current projected video stream is set to the sampling resolution. And in the subsequent display process, the screen projected image of the projected video stream can be scaled and displayed according to the effective resolution.

In order to calculate the contrast value, in some embodiments of the present application, as shown in FIG. 14A , the method further includes:

S1421: Extract black border data by traversing the number of consecutive black pixels in the sampling picture;

S1422: extract initial picture data;

S1423: Calculate the reference value and the comparison value.

In order to calculate the contrast value, it is necessary to determine the resolution of the effective screen in the projected screen. Therefore, as shown in Figure 14B, the range of continuous black areas can be detected from the left side of the image corresponding to the sampled screen, and the range of the black border area can be obtained. : Left side black border width HL , left side black border height _VL _. Then start to detect the range of the continuous black area from the right side of the image, and obtain the range of the black border area: the right black border width _HR and the right black border height _VR , forming black border data. At the same time, the initial picture data can also be extracted from the first frame picture. The initial picture data includes the initial width H ₀ and initial height V ₀ of the first frame picture, so as to calculate the reference value and the contrast value.

Among them, the reference value and the contrast value can be calculated according to the following formula:

The reference value: _S ₀ =V ₀ =V _L =VR ;

The comparative value: S _X =H ₀ -( _{HL +H R} ₎ .

In an implementation manner, as shown in FIG. 15 , before the steps of calculating the reference value and the comparison value, the method further includes:

S14231: Determine whether the sampled picture satisfies a preset execution condition.

S14232: If the sampled picture does not meet the preset judgment condition, set the effective resolution to the initial resolution;

S14233: If the sampled picture satisfies a preset judgment condition, execute the step of calculating the reference value and the comparison value.

In this embodiment, after the sampling picture is acquired, the picture display status in the sampling picture may be first judged to determine whether it satisfies the preset execution condition. The preset execution conditions are: the left black border width _{HL is equal to the right black border width H R} _, _and the left black border height _VL , the right black border height VR and the initial Height V ₀ is equal. That is, it is determined whether or not the sampled picture satisfies " _HL = _HR and _VL ₌ VR= _V0 ".

If the sampling picture does not satisfy " _HL = _HR and _VL ₌ VR =V ₀ ", it is determined that the width and height of the black bars on both sides of the sampling picture are not equal. This situation is generally due to the influence of the display content of the mobile terminal 100B on the sampling result. Since it is difficult to determine the effective resolution in this case, the original resolution can be maintained to output the projected screen, that is, the effective resolution is set to the initial resolution, and the corresponding picture width of the effective resolution H _S =H ₀ , the corresponding picture height V _S =V ₀ .

If the sampled picture satisfies “ _HL = _HR and _VL = _VR =V ₀ ”, the effective resolution can be further judged, that is, the steps of calculating the reference value and the comparison value can be performed to compare the the sampling resolution and the initial resolution.

For example, it is judged whether the sampled picture satisfies "H ₀ -(H _L + _{HR )<V 0 ", if it does not satisfy "H 0 -(H L +H R} ₎ _< _V ₀ _" , then keep the original resolution output, that is The effective resolution is set as the initial resolution, and the corresponding picture width H _S =H ₀ and the corresponding picture height V _S =V ₀ of the effective resolution. If “H ₀ -(H _L +H _R )<V ₀ ” is satisfied, the effective resolution is set as the sampling resolution, and the output picture width corresponding to the effective resolution H _S =H ₀ -(H _L + _HR ), the corresponding picture height V _S =V ₀ .

In practical applications, the content of the screen projected by the mobile terminal 100B easily affects the judgment of the effective screen area in the sampled screen. The range is judged by continuous black pixels on the edge, and the black picture will affect the range judgment of the black area, which in turn affects the final sampling resolution extraction result. Therefore, in order to alleviate the influence of the black picture content on the sampling resolution, the step of extracting the sampling resolution in the projected video stream further includes:

S1201: Acquire multiple frames of sampled images at equal time intervals in the projected screen video stream;

S1202: Calculate the sampling resolution of the sampling picture of each frame respectively.

By presetting the sampling time interval, sampling can be performed multiple times in the screen projection video stream, and the sampling resolution of the picture in each sampling can be extracted separately. For example, a frame of picture image is acquired every T time, and the sampling resolution values obtained by the above-mentioned resolution algorithm are: S _x0 , S _x1 , ..., S _xn .

By performing sampling multiple times, multiple frames of sampling images can be collected along with the changes of the images displayed on the mobile terminal 100B. Usually, the multi-frame sampling pictures are not all affected by the black picture content. Therefore, collecting multi-frame sampling pictures can reduce the influence of the picture content on the judgment of the black border area, thereby improving the accuracy of the effective resolution judgment.

Further, as shown in FIG. 16 , after obtaining the sampling resolutions corresponding to the multi-frame images, the sampling resolutions can also be judged respectively to obtain the comparison results between the sampling resolutions and the initial resolutions, that is, the method further includes: :

S1211: Compare the sampling resolution of each frame of the sampling picture with the initial resolution respectively;

S1212: If all the sampling resolutions are greater than or equal to the initial resolution, set the effective resolution of the video stream as the initial resolution;

S1213: If all the sampling resolutions are smaller than the initial resolution, set the effective resolution of the video stream as the sampling resolution.

By comparing the sampling resolution and the initial resolution of each frame of sampling images respectively, if consecutive S _x0 ≤S ₀ , S _x1 ≤S ₀ , ..., S _xn ≤S ₀ , the effective resolution is obtained as S _xn , and set S ₀ =S _xn . If consecutive S _x0 ≥S ₀ , S _x1 ≥S ₀ , ..., S _xn ≥S ₀ , the effective resolution is obtained as S ₀ .

In an implementation manner, the method further includes: if the effective resolution of the video stream is set to be the sampling resolution, then controlling to rotate the display 275 of the display device 200 to a vertical screen state. For example, the initial resolution is 1920×1080 and the sampling resolution is 960×1080. By calculating the reference value and the contrast value, the width of the effective picture in the current sampling picture can be determined S _X =H ₀ -(H _L +H _R )=960, and the height of the first frame picture is 1080, so the current effective resolution can be determined is the sampling resolution, that is, the resolution of the effective picture: 960×1080. Therefore, it can be determined that the corresponding display picture on the mobile terminal 100B is a vertical picture of 960×1080.

The vertical screen is more suitable for displaying in the vertical screen state. Therefore, after determining that the effective resolution is the sampling resolution, the controller 250 can send a control command to the rotating component 276, so that the rotating component 276 drives the display 275 counterclockwise (or clockwise). Hour hand) rotate to portrait orientation.

After the display 275 is rotated to the vertical screen state, the screen projection image can be displayed according to the aspect ratio of 960:1080. However, because the screen of the display 275 is relatively large, the display resolution is usually 3840×2160 (in a horizontal screen state, and 2160×3840 in a corresponding vertical screen state). Therefore, in order to display a screen projection image with a resolution of 960×1080, the screen projection image needs to be scaled, so that the display 275 can completely display the screen projection image.

In addition, due to the simple adjustment of the size of the projected screen image, the displayed projected screen image is likely to be blurred on a large screen, which seriously reduces the user experience. Therefore, while zooming the projected screen image, the image quality adjustment related to pixel interpolation can also be performed on the projected screen image, so as to improve the blurred image and improve the image display effect.

In another exemplary embodiment of the present application, another method for processing screen projection images is provided, and specifically another display device is provided, including:

monitor;

a user interface, the communication is configured to connect to a terminal;

And the controller, see Figure 17, is configured to execute:

S11', receiving image information sent by the terminal, when the terminal is in a portrait mode, the image information includes valid information and left and right black area information.

After performing the screen projection operation, the mobile terminal 100B will send the displayed picture to the display device 200 through a screen projection protocol, such as Miracast protocol or other screen projection and mirroring protocols. As new interactive pictures are continuously generated during the screen projection process, the mobile terminal 100B will send the pictures to the display device 200 frame by frame to form a screen projection video stream.

S12', if the current rotation state of the display does not match the portrait mode of the terminal, rotate the display to a portrait state;

In practical applications, it can be determined whether the current rotation state of the display matches the display state of the terminal by analyzing whether the image information contains left and right black information respectively.

In some exemplary implementations, the controller acquires the rotation angle callback information of the display, and determines the target rotation state of the display according to whether the image information acquired from the mobile terminal includes left and right black borders.

When the image information does not contain left and right black information, it means that the mobile terminal is currently in the landscape mode, and when the current rotation state of the detected display is the landscape state, the two are matched. No need to rotate the TV. When it is detected that the current optional state of the monitor is the portrait state, the two do not match, and the monitor needs to be rotated to the landscape state.

When the image information contains left and right black areas, it means that the mobile terminal is currently in portrait mode, and when the current rotation state of the display is detected to be landscape, the two do not match, and the display needs to be rotated to portrait. When the current optional state of the detection monitor is vertical, the two are matched, and the monitor does not need to be rotated.

After the user performs the screen projection operation through the mobile terminal 100B, the mobile terminal 100B sends the screen projection image to the display device 200 through the mirroring protocol or the screen projection protocol. The controller 250 can receive the image information sent by the terminal, and detect the current rotation state of the display 275 . Wherein, the detection of the rotation state of the display 275 can be completed by a sensor built in the display device 200 .

For example, sensor devices such as a gyroscope and a gravitational acceleration sensor may be provided on the display 275 of the display device 200 to determine the attitude data of the display 275 relative to the gravitational direction by measuring angular acceleration or gravitational direction. Then, the detected attitude data is compared with the attitude data in the horizontal screen state and the vertical screen state, respectively, to determine the current rotation state of the display 275 . For another example, a grating angle sensor, a magnetic field angle sensor or a sliding resistance angle sensor, etc. can be provided on the rotating component 276, and the angle rotated by the rotating component 276 is measured and compared with the angles in the horizontal screen state and the vertical screen state, respectively, to determine The rotation state that the display 275 is currently in.

In some exemplary embodiments, the controller is further configured to:

Calculate the rotation direction and rotation angle of the screen projection image; the rotation direction of the screen projection image is opposite to the rotation direction of the display; the rotation angle of the screen projection image is equal to the rotation angle of the display;

Rotate the projection screen according to the rotation direction and the rotation angle.

S13', controlling the display to present a screen projection image based on the valid information, wherein the screen projection image is obtained by enlarging the valid information by a preset multiple.

After the display device 200 receives the screen projection video stream, the controller 250 of the display device 200 may perform frame-by-frame analysis on the received screen projection video stream. For example, the aspect ratio of the screencast video stream sent by the mobile terminal 100B is 1920:1080. After receiving the screencast video stream, the controller 250 can obtain frame images by parsing the screencast video stream. The resolution of the extracted frame image is 1920×1080, which is the initial resolution.

After the resolution is extracted, the frame image in the projected video stream can be sampled again to extract the effective resolution. The frame of picture used for sampling is called a sampling picture. The effective resolution is the resolution of the effective picture on the frame data extracted from the screen projection video stream. Specifically, the effective resolution may be obtained from the screen-casting video stream at preset time intervals.

During the sampling process, in order to determine the valid picture from the sampled picture, the pixel color of the sampled picture can be traversed. Obviously, the pixel color value of the black area is black, and the pixel color value of the valid area is usually not all black. Therefore, it can be determined by traversing each pixel of the sampled image that the black and rectangular area is a black border, and other areas are valid. screen.

It should be noted that, according to different adaptive display methods of the display device 200, the color filled in the black area is not limited to black only. For example, in order to adapt to the overall UI design style of the operating system, the black area can be gray, blue, or other colors, as well as gradient colors, specific patterns, etc. For these cases, in this application, for the convenience of subsequent description, they are still referred to as black areas or black borders.

After obtaining the frame image resolution and effective resolution for the projected video stream, the effective resolution can be compared with the frame image resolution to determine the current The status of the valid images in the projected video stream (such as scale, orientation, etc.), so as to choose whether to display according to the valid images.

For example, if the effective resolution is equal to the frame image resolution, that is, the resolution of the first frame is 1920×1080, and the effective area resolution determined in the sampled image is also 1920×1080, it means that the current projection screen does not exist With black borders, the projected screen can fill the display area. That is, the display requirements of the screen projection image can be satisfied by directly displaying the screen in the horizontal screen state of the display 275 .

It should be noted that, because the resolution of the display screen is usually represented by the number of pixels occupied in the width and height directions of the screen, for example, 1920×1080. However, it is usually difficult to directly compare the value of resolution alone. For example, from a numerical comparison, a resolution of 1920×1080 is equal to 1080×1920. Therefore, in the actual comparison process, the comparison between the effective resolution and the frame image resolution can be obtained by extracting part of the value in the resolution or converting the resolution into other comparable values, and then performing the comparison. result. For example, the width or height of the overall picture can be extracted from the frame image resolution and compared with the height or width of the effective picture to determine its effective resolution.

It can be seen from the above technical solutions that the method for detecting the effective resolution of a screencast video stream provided by the present application can extract the frame image resolution and the effective resolution from the screencast video stream after receiving the screencast video stream, and compare them to determine the current resolution. The effective resolution of the video stream. By setting the effective resolution of the video stream, the projected screen can be displayed according to the effective resolution, so as to adapt to the display direction of the projected screen, reduce the influence of black borders, and achieve a better user experience.

In order to calculate the magnification of the effective picture, in some embodiments of the present application, as shown in FIG. 18 , the method further includes:

S1131: Extract black border data by traversing the number of consecutive black pixels in the sampling picture;

S1132: extract initial picture data;

_S1133 : Calculate the height ratio _RH and width ratio RW of the display device and the effective screen;

S1134: Determine the magnitude of R _H > R _W.

As shown in FIG. 14 , the width of the image frame transmitted from the mobile terminal is a or denoted as H ₀ , and the height is H _ph or denoted as _VL ; the width of the display device in the vertical screen state is W _tv , and the height is H _tv . The range of the continuous black area can be detected from the left side of the image corresponding to the effective picture, and the range of the black area can be obtained: the left black border width a and the left black border height H _ph . Then start to detect the range of the continuous black area from the right side of the image, and obtain the range of the black area: the right black border width a and the right black border height H _ph , forming black border data. At this time, in the obtained image information sent by the mobile terminal, the resolution of the effective picture is (H ₀ -2a)*V _Lh . The resolution of the display device is W _tv *H _tv .

The height ratio of the display device and the effective information _RH =H _tv /H _ph ; the width ratio RW = _W ₀ /( _W ₀ -2a), if _RH >RW , then execute S1135, and the effective information Enlarge RW times to obtain a screen projection image; if R _H < _{R W} _, execute S1136, and magnify the valid information by _RH times to obtain a screen projection image.

It can be seen from the above technical solutions that, as shown in FIG. 19 , the display device provided by the present application can determine whether the mobile terminal is in portrait mode according to the image information, and automatically adjust the rotation state of the display, so as to use a larger display space to display the projected screen, Alleviates the problem that traditional smart TVs cannot display the projected screen normally.

At present, there are many protocols for screen projection and mirroring, such as the Miracast standard-based screen projection protocol, the DLNA standard-based screen projection protocol, the Airplay standard-based screen projection protocol, or the customizable screen projection method.

Among them, in some screen projection protocols, the display device can obtain the real video stream of the mobile terminal. For example, the Airplay protocol can send the image data of the screen of the mobile terminal itself to the display device; in this way, the display device can directly Determines whether the image is landscape or portrait.

In other screen projection protocols, the display device cannot obtain the real screen video stream of the mobile terminal, that is, the mobile device first processes the screen data information before transmitting the screen projection data to the display device, such as the Miracast standard screen projection protocol. , the smart device always sends horizontal media resources to the display device, whether the screen is placed horizontally or vertically, the display device always obtains horizontal video data.

For example, when the mobile terminal is placed vertically for mirroring, before sending the screen data to the display device, black borders are added on both sides of the screen data. Its original intention is to adapt to display devices placed in landscape orientation.

However, after the display device can be rotated horizontally and vertically, the current Miracast standard screencasting protocol does not adapt to the vertical screen display device for the time being, and still sends horizontal screen resources with black borders to the display device.

In the embodiment shown next, when the display device receives the screen projection resource, it judges whether the image has black borders. Therefore, the display device cannot know the actual placement of the mobile terminal. In this case, the display device will not be able to rotate according to the active frame in the image. That is, regardless of whether the mobile phone has a horizontal screen or a vertical screen, after receiving the horizontal screen projection data, the screen projection data is directly displayed.

Of course, the display device just does not rotate according to the screen projection data. Regardless of whether the display device is in a landscape or portrait state, the user can still actively issue a rotation command to rotate the display device to a different state than before. . Viewed from the front of the display device, the rotating component 276 can rotate the display screen to a vertical screen state, that is, a state in which the vertical side length of the screen is greater than the horizontal side length, or can rotate the screen to a landscape screen state, that is, the screen is in a horizontal screen state. A state where the side length is greater than the vertical side length.

For example, if the display device is in a landscape position, the screen shown in FIG. 10A or 11A is displayed. For FIG. 10A , the mobile terminal is placed horizontally, and the screen projection data stream is a full-screen horizontal resource with no black borders on both sides. For FIG. 11A , the mobile terminal is placed vertically, and the screen projection data stream is a horizontal resource in which the effective picture with black borders on both sides is only in the middle of the data stream. In these two display placements, for the display device, it receives all horizontal and vertical screen resources.

If the display device is in a vertical position, the screen shown in FIG. 10B or 11B is displayed. In the case of FIG. 11B , since the effective screen occupies a small proportion of the screen, in order to improve the user experience, in the exemplary embodiment of the application, a control command can be issued through a remote control, voice, gesture, etc., to control the current mirroring interface. Zooming in, where appropriate, on the user interface of the display device, a zoom scale is shown, as shown in Figure 20A. After a zoom-in operation is received, optionally, the operation bar automatically disappears between presets. For example, the remote control can be used to call up, down, left, and right, and the remote control can be manually enlarged to an appropriate multiple, as shown in FIGS. 20B and 20C .

In some embodiments, the maximum value of the maximum magnification corresponds to a full-screen display of an effective picture.

In a specific implementation, the present application also provides a non-volatile computer storage medium, wherein the computer storage medium can store a program, and when the program is executed, it can include some or all of the steps in the various embodiments of the method provided by the present application. , when the controller of the display device provided by the present application executes the computer program instructions, the controller executes the steps in which the controller described in the present application is configured. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), and the like.

Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions in the embodiments of the present application can be embodied in the form of software products in essence or in the parts that make contributions to the prior art, and the computer software products can be stored in a storage medium, such as ROM/RAM , magnetic disk, optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present application.

It is sufficient to refer to each other for the same and similar parts among the various embodiments in this specification. In particular, as for the embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant part may refer to the description in the method embodiment.

The above-described embodiments of the present application do not limit the protection scope of the present application.

Claims

A display device comprising:

monitor;

a rotation component, configured to drive the display to rotate, so that the display is in a rotation state of a landscape screen state or a portrait screen state;

a user interface, the communication is configured to connect to a terminal;

Controller, configured as:

Receive screen projection media resources, and determine the attributes of the media resources;

If the media resource attribute is video, detect the horizontal and vertical screen status of the display, obtain the video aspect ratio, and determine the matching situation between the video aspect ratio and the horizontal and vertical screen status of the display, and determine the matching situation according to the matching Circumstances determine whether the display needs to be rotated;

If the media resource attribute is a picture, the screen is filled and displayed according to the original aspect ratio of the current image.
The display device of claim 1, the controller is further configured to:

When the if the video aspect ratio does not match the horizontal and vertical screen states of the display, the display is driven to rotate. When the aspect ratio of the video matches the horizontal and vertical screen states of the display, it is directly displayed on the display to adapt the screen according to the original ratio to present the video content.
The display device of claim 2, wherein the controller executes to drive the display to rotate if the video aspect ratio does not match the horizontal and vertical screen states of the display, and is further configured to:

Obtain the aspect ratio of the video and adapt it to the entire screen according to the original ratio on the monitor for display and playback;

After the display is rotated so that the video aspect ratio is adapted, the video picture is displayed in full screen.
The display device of claim 2, wherein the controller executes to drive the display to rotate if the video aspect ratio does not match the horizontal and vertical screen states of the display, and is further configured to:

obtaining the original resolution of the video that is adapted to the display before rotation;

Determine the zoom factor of the video picture according to the original resolution and the full screen resolution of the display;

Monitor the rotation angle information of the display, and gradually scale the video image according to the rotation angle information to ensure that the video image is finally displayed in full screen.
The display device of claim 4, the controller is further configured to:

If the initial resolution is F1 and the full screen resolution is F2, the video image needs to be enlarged by F2/F1 times during the rotation process. After listening to the rotation angle of the display from landscape to portrait, the entire rotation process can be divided into 90 parts according to the rotation angle, and each degree of rotation is enlarged (F2/F1)/90.
A method for receiving screencast content, applied to a display device, includes:

Receive screen projection media resources, and determine the attributes of the media resources;

If the media resource attribute is video, detecting the horizontal and vertical screen states of the display, obtaining the video aspect ratio, and judging whether the video aspect ratio matches the horizontal and vertical screen states of the display;

If the media resource attribute is a picture, the screen is filled and displayed according to the original aspect ratio of the current image.
The method for receiving screencast content according to claim 6, comprising:

When the if the video aspect ratio does not match the horizontal and vertical screen status of the display, drive the display to rotate, and when the video aspect ratio matches the horizontal and vertical screen status of the display, it is directly displayed on the display The video content is displayed by adapting the screen according to the original ratio.
The method for receiving screen projection content according to claim 7, wherein the driving the display to rotate if the video aspect ratio does not match the horizontal and vertical screen states of the display, further comprising:

Obtain the aspect ratio of the video and adapt it to the entire screen according to the original ratio on the monitor for display and playback;

After the display is rotated so that the video aspect ratio is adapted, the video picture is displayed in full screen.
The method for receiving screen projection content according to claim 7, wherein the driving the display to rotate if the video aspect ratio does not match the horizontal and vertical screen states of the display, further comprising:

obtaining the original resolution of the video that is adapted to the display before rotation;

Determine the zoom factor of the video picture according to the original resolution and the full-screen resolution of the display;

Monitor the rotation angle information of the display, and gradually scale the video image according to the rotation angle information to ensure that the video image is finally displayed in full screen.
A non-volatile storage medium performing the method for performing the above claims 6-9.
A display device comprising:

monitor;

a rotation component, configured to drive the display to rotate, so that the display is in a rotation state of a landscape screen state or a portrait screen state;

a user interface, the communication is configured to connect to a terminal;

Controller, configured as:

receiving image information sent by the terminal, wherein, when the terminal is in a portrait mode, the image information includes valid information and left and right black information, and the valid information corresponds to the screen display content of the terminal;

If the current rotation state of the display does not match the portrait mode of the terminal, rotate the display to the portrait state;

The display is controlled to present a screen projection image based on the valid information, wherein the screen projection image is obtained by amplifying the valid information by a preset multiple.
The display device of claim 11, the controller is further configured to:

If the current rotation state of the display matches the portrait mode of the terminal, the display is controlled to present a screencast image based on the valid information.
The display device according to claim 11, wherein the controller determines whether the current rotation state of the display matches the portrait mode of the terminal, and is further configured to:

Obtain the display rotation angle callback information;

A target rotation state of the display is determined based on the image information.
The display device of claim 13, wherein the controller determines a target rotation state of the display according to the image information, further configured to:

If it is received that the image information sent by the terminal includes left and right black information, it is determined that the target rotation state of the display is a vertical screen state;

If it is received that the image information sent by the terminal does not include left and right black information, it is determined that the target rotation state of the display is a landscape state.
The display device according to claim 11, wherein the controller controls the display to present a screen projection image based on the valid information, wherein the screen projection image is obtained by enlarging the valid information by a preset multiple, and is further Configured as:

Calculate the height ratio RH and width ratio RW of the display device and the effective information,

If R H > R W , then the effective information is enlarged by R W times to obtain a screen projection image;

If R H < R W , the effective information is enlarged by R H times to obtain a screen projection image.
The display device of claim 13, the controller is further configured to:

Calculate the rotation direction and rotation angle of the screen projection image; the rotation direction of the screen projection image is opposite to the rotation direction of the display; the rotation angle of the screen projection image is equal to the rotation angle of the display;

Rotate the projection screen according to the rotation direction and the rotation angle.
A screen projection method, applied to a display device, includes:

receiving image information sent by the terminal, wherein, when the terminal is in a portrait mode, the image information includes valid information and left and right black information, and the valid information corresponds to the screen display content of the terminal;

If the current rotation state of the display does not match the portrait mode of the terminal, rotate the display to the portrait state;

The display is controlled to present a screen projection image based on the valid information, wherein the screen projection image is obtained by amplifying the valid information by a preset multiple.
The screen projection method according to claim 17, comprising:

If the current rotation state of the display matches the portrait mode of the terminal, the display is controlled to present a screen projection image based on the valid information, wherein the screen projection image is obtained by magnifying the valid information by a preset multiple.
The screen projection method according to claim 17, wherein the determining whether the current rotation state of the display matches the portrait mode of the terminal comprises:

Obtain the display rotation angle callback information;

A target rotation state of the display is determined based on the image information.
The screen projection method according to claim 19, wherein the determining the target rotation state of the display according to the image information comprises:

If it is received that the image information sent by the terminal includes left and right black information, determine that the target rotation state of the display is a vertical screen state;

If it is received that the image information sent by the terminal does not include left and right black information, it is determined that the target rotation state of the display is a landscape state.