WO2023240973A1

WO2023240973A1 - Display device and screen-projection method

Info

Publication number: WO2023240973A1
Application number: PCT/CN2022/140843
Authority: WO
Inventors: 陈效总; 佘祥顺; 赖园园; 薛梅; 庞秀娟; 范传恒; 李鑫; 张凯
Original assignee: 聚好看科技股份有限公司
Priority date: 2022-06-16
Filing date: 2022-12-21
Publication date: 2023-12-21

Abstract

Provided in the present application are a display device, a terminal device and a screen-projection method. The display device comprises a display and a controller connected to the display, wherein the controller is configured to display a follow-up interface on the display in response to a received starting instruction, wherein the follow-up interface comprises a first window and a second window, the first window is used for playing a training video stream acquired from a server, and the second window is used for playing a follow-up video stream received from a terminal device.

Description

Display equipment and screencasting methods

This application requires application numbers 202210743842.0 submitted on June 27, 2022, application numbers 202210689400.2 submitted on June 16, 2022, 202210767185.3 submitted on June 30, 2022, and 202210767185.3 submitted on June 29, 2022. The priority of Chinese patent application with application number 202210764007.5, the entire content of which is incorporated into this disclosure by reference.

Technical field

The embodiments of this application relate to the field of display technology. More specifically, it relates to a display device and a screen projection method.

Background technique

Nowadays, TVs provide more and more functions, and gradually begin to support access through mobile phones as camera equipment to complete product functions that require shooting. With the development of terminal technology and Internet technology, multi-screen collaboration technology has been widely used. Multi-screen collaboration technology refers to the collaborative work of multiple screens, which includes projecting the screen content of one device to other devices for display. For example, the screen content of the mobile phone is projected to the computer for display, so that the user can watch the content displayed on the screen of the mobile phone on the computer, thus realizing the same screen display between the mobile phone and the computer.

Contents of the invention

Some embodiments of the present application provide a display device, including: a display; and a controller configured to: in response to a received start instruction, display a follow-up training interface on the display, wherein the follow-up training interface includes a first window and a second window, the first window is used to play the training video stream obtained from the server, and the second window is used to play the follow-up video stream received from the terminal device.

Some embodiments of the present application provide a screen projection method, including: in response to a received startup instruction, displaying a follow-up training interface on the display of the display device, wherein the follow-up training interface includes a first window and a second window, so The first window is used to play the training video stream obtained from the server, and the second window is used to play the follow-up video stream received from the terminal device.

Description of the drawings

Figure 1A is a scene architecture diagram of a screen casting method according to some embodiments;

Figure 1B is a system network architecture diagram of a communication message processing method according to some embodiments;

Figure 2 is a hardware configuration block diagram of a control device according to some embodiments;

Figure 3A is a hardware configuration block diagram of a display device or terminal device according to some embodiments;

Figure 3B is a hardware configuration block diagram of a mobile terminal according to some embodiments;

Figure 4A is a software configuration diagram of a display device or mobile terminal according to some embodiments;

Figure 4B is a schematic diagram of software interaction between a mobile terminal and a display device implementing a follow-up page control method according to some embodiments;

Figure 5 is a step flow chart of a screen projection method according to some embodiments;

Figure 6 is a step flow chart of a screen projection method according to other embodiments;

Figure 7 is a schematic diagram of the scene of the screen projection method shown in Figure 6;

Figure 8 is a schematic diagram of the screen projection method shown in Figure 6;

Figure 9 is a step flow chart of a screen projection method according to other embodiments;

Figure 10 is a schematic diagram of the screen projection method shown in Figure 9;

Figure 11 is a step flow chart of a screen projection method according to other embodiments;

Figure 12 is a schematic diagram of the scene of the screen projection method shown in Figure 11;

Figure 13 is a schematic diagram of the screen projection method shown in Figure 11;

Figure 14 is a step flow chart of a screen projection method according to other embodiments;

Figure 15 is a schematic diagram of the screen projection method shown in Figure 14;

Figure 16 is a step flow chart of a screen projection method according to other embodiments;

Figure 17 is a schematic diagram of the scene of the screen projection method shown in Figure 16;

Figure 18 is a schematic diagram of the screen projection method shown in Figure 16;

Figure 19 is a scene architecture diagram of a screencasting method according to some embodiments;

Figure 20 is a schematic flowchart of a method for controlling a follow-up page according to some embodiments;

Figure 21 is a schematic diagram of a scene displaying a follow-up page according to some embodiments;

Figure 22 is a schematic diagram of displaying power data in a follow-up training page according to some embodiments;

Figure 23 is a schematic flowchart of another follow-up page control method according to some embodiments;

Figure 24 is a schematic diagram of displaying prompt information in a follow-up page according to some embodiments;

Figure 25 is a schematic flowchart of another follow-up page control method according to some embodiments;

Figure 26 is a schematic diagram of prompting to terminate follow-up training on the follow-up training page according to some embodiments;

Figure 27 is an interactive schematic diagram of a follow-up page control method according to some embodiments;

Figure 28 is an interactive schematic diagram of another follow-up page control method according to some embodiments;

Figure 29A is a schematic flowchart of a communication message processing method according to some embodiments;

Figure 29B is a schematic signal flow diagram of a communication message processing method according to some embodiments;

Figure 30A is a schematic flowchart of another communication message processing method according to some embodiments;

Figure 30B is a schematic signal flow diagram of another communication message processing method according to some embodiments;

Figure 31A is a schematic flowchart of yet another communication message processing method according to some embodiments;

Figure 31B is a schematic signal flow diagram of yet another communication message processing method according to some embodiments;

Figure 32A is a schematic flowchart of yet another communication message processing method according to some embodiments;

Figure 32B is a schematic signal flow diagram of yet another communication message processing method according to some embodiments;

Figure 33A is a schematic flowchart of yet another communication message processing method according to some embodiments;

Figure 33B is a schematic signal flow diagram of yet another communication message processing method according to some embodiments;

Figure 34A is a schematic flowchart of yet another communication message processing method according to some embodiments;

Figure 34B is a schematic signal flow diagram of yet another communication message processing method according to some embodiments;

Figure 35A is a schematic flowchart of yet another communication message processing method according to some embodiments;

Figure 35B is a schematic signal flow diagram of yet another communication message processing method according to some embodiments;

Figure 36 is a schematic structural diagram of an electronic device according to some embodiments;

Figure 37A is a schematic flowchart of a screen projection method according to some embodiments;

Figure 37B is a schematic signal flow diagram of a screen projection method according to some embodiments;

Figure 37C is an interactive schematic diagram according to some embodiments;

Figure 38A is a schematic flowchart of another screen projection method according to some embodiments;

Figure 38B is a schematic signal flow diagram of another screen projection method according to some embodiments;

Figure 38C is a schematic flowchart of yet another screencasting method according to some embodiments;

Figure 38D is a schematic signal flow diagram of yet another screen projection method according to some embodiments;

Figure 39A is a schematic flowchart of yet another screen projection method according to some embodiments;

Figure 39B is a schematic signal flow diagram of yet another screen projection method according to some embodiments.

Detailed ways

Some embodiments of the present application will be described below with reference to the accompanying drawings in some embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments.

In related fields, the solution to achieve the same camera function on the TV side is to interconnect the mobile phone with the TV, collect the video through the camera of the mobile phone, and project the collected images to the TV for display. However, the aspect ratios of TV and mobile phone screens are generally very different, and mobile phones can capture videos in various postures such as horizontal screen recording and vertical screen recording. Therefore, the projected videos sent by mobile phones are often different from those used for playing projections on TVs. The video playback window of the screen video does not match, which leads to problems such as incomplete video images and imbalanced proportions, seriously affecting the user experience. Therefore, when the TV receives the screen projection video stream sent by the mobile phone, how to adjust the screen projection video stream so that the screen projection video stream can adapt to the video playback window is an urgent problem to be solved.

According to some embodiments of the present application, a display device and a screen projection method are provided to solve the problem that the projected video sent by the mobile phone may not match the video playback window used to play the screen projection video on the TV.

FIG. 1A is a schematic diagram of the scene architecture of the screen casting method provided by the embodiment of the present application. As shown in Figure 1A, the scene architecture provided by the embodiment of the present application includes: a control device 100, a display device 200, a terminal device 300, and a server 400. In some embodiments, the display device may be in data communication with server 400.

The user can operate the display device 200 through the terminal device 300 or the control device 100, and can send a video stream to the display device 200 through the terminal device 300 to control the display device 200 to perform corresponding operations, and perform operations according to the video stream sent by the display device 200. Video playback. The display device provided by the embodiment of the present application can have a variety of implementation forms, such as a television, a smart speaker refrigerator with a display function, a curtain with a display function, a personal computer (Personal Computer, PC), a laser projection device, a monitor (monitor) ), electronic whiteboard (electronic bulletin board), wearable devices, vehicle-mounted equipment, electronic desktop (electronic table), etc.

In some embodiments, the control device 100 may be a remote controller. The communication between the remote controller and the display device 200 includes infrared protocol communication or Bluetooth protocol communication, and other short-distance communication methods to control the display device 200 through wireless or wired methods. The user can control the display device 200 by inputting user instructions through buttons on the remote control, voice input, control panel input, etc.

In some embodiments, the display device 200 can also be controlled using a terminal device 300 (such as a mobile terminal, a tablet, a computer, a laptop, an AR/VR device, etc.). For example, the display device 200 is controlled using an application program running on the terminal device.

In some embodiments, the terminal device 300 and the display device may also be used to communicate data.

In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the terminal device 300 . For example, the display device 200 can directly receive the user's voice command control through a module configured inside the display device 200 to obtain voice commands.

In some embodiments, the display device 200 can also perform data communication with the server 400 to obtain related media resources from the server. The display device 200 may be allowed to communicate with the server 400 through a local area network (LAN) or a wireless local area network (WLAN). The server 400 can provide media resource services and various contents and interactions to the display device 200 .

FIG. 2 is a block diagram illustrating the configuration of the control device 100 in the embodiment shown in FIG. 1 . As shown in Figure 2, the control device 100 includes a controller 110, a communication interface 130, a user input/output interface 140, a memory, and a power supply. The control device 100 can receive the operation instructions input by the user, convert the operation instructions into instructions that the display device 200 can recognize and respond to, and forward the operation instructions or instructions obtained by converting the voice instructions to the display device 200 to interact between the user and the display device 200 The intermediary role between.

As shown in Figure 3A, the display device 200 or the terminal device 300 includes a tuner and demodulator 210, a communicator (also called a communication interface) 220, a detector 230, an external device interface 240, a controller 250, a display 260, and an audio output interface 285. , at least one of memory, power supply, and user input interface 280 .

In some embodiments, the controller 250 includes a processor, a video processor, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input/output.

The detector 230 is used to collect signals from the external environment or interactions with the outside. For example, the detector 230 includes a light receiver, a sensor used to collect ambient light intensity; or the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures, or , the detector 230 includes a sound collector, such as a microphone, etc., for receiving external sounds.

The external device interface 240 may include, but is not limited to, any one of the following: high-definition multimedia interface (HDMI), analog or data high-definition component input interface (component), composite video input interface (CVBS), USB input interface (USB), RGB port, etc., or Multiple interfaces. It can also be a composite input/output interface formed by the above-mentioned multiple interfaces.

The user may input a user command into a graphical user interface (GUI) displayed on the display 260, and the user interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user interface recognizes the sound or gesture through the sensor to receive the user input command.

In some embodiments, the user input interface 280 is an interface that can be used to receive control input (such as physical buttons on the display device body, or others).

Referring to Figure 4A, in some embodiments, the operating system of the display device 200 is divided into four layers. From top to bottom, they are the Applications layer (referred to as the "Application layer") and the Application Framework layer. (referred to as "framework layer"), Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and kernel layer.

As shown in Figure 4A, the application framework layer in the embodiment of the present application includes managers (Managers), content providers (Content Provider), etc., where the manager includes at least one of the following modules: Activity Manager (Activity Manager) Interact with all activities running in the system; Location Manager is used to provide system services or applications with access to system location services; Package Manager is used to retrieve files currently installed on the device Various information related to the application; Notification Manager is used to control the display and clearing of notification messages; Window Manager is used to manage icons, windows, toolbars, wallpapers and desktops on the user interface part.

In some embodiments, the controller 250 displays a follow-up interface on the display 260 in response to the received trigger instruction (also called a start instruction), where the follow-up interface includes a first video playback window and a third video playback window. Two video playback windows (also known as the first window and the second window), the second video window is used to play the fitness video obtained from the server;

In some embodiments, when the controller 250 receives the first video stream generated by the terminal device based on its video recording device sent by the terminal device through the communicator 220, the controller 250 uses the identification information, video playback parameters and The size of the first video stream is used to obtain a scaling ratio, the first video stream is scaled according to the scaling ratio to obtain a second video stream, and the layout is obtained according to the video playback parameters and the size of the second video stream. Offset parameter: offset the second video stream based on the layout offset parameter, and control the display 260 to play the offset second video stream in the first video window.

In some embodiments, the first video window is used to play the fitness video obtained from the server, and the second video window plays the offset second video stream.

In some embodiments, the identification information includes: a first gesture identification or a second gesture identification; when the identification information includes a first gesture identification, the identification information is used to identify the terminal device to record the first gesture. The posture when recording the video corresponding to the video stream is the portrait posture; when the identification information includes a second posture identification, the identification information is used to identify the posture when the terminal device records the video corresponding to the first video stream. Horizontal posture. Wherein, the video playback parameters corresponding to the first gesture identification and the video playback parameters corresponding to the second gesture identification are different.

In some embodiments, when the identification information is used to identify that the posture of the terminal device when recording the video corresponding to the first video stream is a portrait posture, the controller 250 adjusts the posture of the terminal device according to the scaling ratio. The first video stream is scaled to obtain the second video stream, and the layout offset parameter is obtained according to the video playback parameter and the size of the second video stream, including: the controller 250 according to the width of the first video stream. and the target width to obtain the first scaling ratio; the target width is the width of the first video playback window or the width of the preset video stream size;

Scale the first video stream using the first scaling ratio to obtain the second video stream; obtain a first layout offset according to the height of the second video stream and the height of the first video playback window parameter;

The controller 250 offsets the second video stream based on the layout offset parameter, including the controller offsets the second video stream in a height direction based on the first layout offset parameter. shift.

Wherein, the first scaling ratio is the ratio of the target width to the width of the first video stream; the first layout offset parameter is the height of the second video stream to the first video playback window. half the difference in height.

In some embodiments, when the identification information is used to identify that the posture of the terminal device when recording the video corresponding to the first video stream is a horizontal screen posture,

The controller 250 obtains a scaling ratio according to the identification information, video playback parameters and the size of the first video stream, and scales the first video stream according to the scaling ratio to obtain a second video stream, including:

The controller 250 obtains the second scaling ratio according to the height of the first video stream and the target height; the target height is the height of the first video playback window or the height in the preset video stream size; using the The second scaling ratio scales the first video stream to obtain the second video stream;

The controller 250 obtains the layout offset parameter according to the video playback parameter and the size of the second video stream, including: the controller 250 obtains the layout offset parameter according to the width of the second video stream and the width of the first video playback window. , obtain a second layout offset parameter; and offset the second video stream in the width direction based on the second layout offset parameter. Wherein, the second scaling ratio is the ratio of the target height to the height of the first video stream; the second layout offset parameter is the width of the second video stream and the first video play window half the difference in width.

In some embodiments, the communicator 250 offsets the second video stream based on the layout offset parameter, and controls the display to play the offset second video in the first video window. After streaming, the controller 250 is also configured to:

In response to the received trimming and playback instruction, a third zoom ratio is obtained according to the size of the selected area and the size of the first video playback window; the trimming and playback instruction carries the size of the selected area and the size of the selected area. The position information of a certain area is used to indicate video playback based on the video content of the first video stream in the selected area; the third scaling ratio is the smaller of the first ratio and the second ratio, The first ratio is the ratio of the height of the first video playback window to the height of the selected area, and the second ratio is the ratio of the width of the first video playback window to the width of the selected area. ratio;

Use the third zoom ratio to zoom the first video stream to obtain a third video stream; according to the third zoom ratio, the width of the first video playback window, the position information of the selected area and the Obtain the third layout offset parameter based on the width of the selected area; obtain the third layout offset parameter according to the third scaling ratio, the height of the first video playback window, the position information of the selected area, and the height of the selected area. Four layout offset parameters;

Video playback is performed in the first video playback window based on the third video stream, and the third video stream is offset in the width direction based on the third layout offset parameter. Based on the fourth The layout offset parameter offsets the third video stream in the height direction, and controls the display to play the offset third video stream in the first video window.

In some embodiments, the controller 250 obtains a third layout bias according to the third zoom ratio, the width of the first video playback window, the position information of the selected area, and the width of the selected area. Shift parameters, including:

The controller 250 calculates the third layout offset parameter through the following formula 1:

The first formula is: CLIP ₃ = (X+select_w/2)*R3-WL/2;

The controller 250 obtains a fourth layout offset parameter according to the third zoom ratio, the height of the first video playback window, the position information of the selected area, and the height of the selected area, including: The controller 250 calculates the third layout offset parameter through the following formula 2:

The second formula is: CLIP ₄ = (Y+select_h/2)*R3-HL/2;

Wherein, CLIP ₃ is the third layout offset parameter, CLIP ₄ is the fourth layout offset parameter, and X is the distance from the left edge of the selected area to the left edge of the first video stream. , Y is the distance from the top edge of the selected area to the top edge of the first video stream, select_w/2 and select_h/2 are the width and height of the selected area respectively, and R3 is the Three scaling ratios, WL and HL are the width and height of the first video playback window respectively.

Figure 5 is a schematic flowchart of a screen casting method provided according to some embodiments of the present application. As shown in Figure 5, the screen casting method provided by embodiments of the present application includes the following steps:

S51. Receive the first video stream sent by the terminal device.

Wherein, the first video stream is a video stream obtained through video collection by a video recording device of the terminal device, and carries an identifier used to identify the posture of the terminal device when recording the video corresponding to the first video stream. information.

In some embodiments, the first video stream may be a video stream that is recorded in real time by the terminal device through a video recording device and sent to the display device. In other embodiments, the first video stream may also be a video stream sent to the display device when the terminal device displays the pre-recorded video on the screen.

In some embodiments, the terminal device in the embodiment of the present application may be a mobile phone, and the video recording device of the terminal device may be a front/rear camera of the mobile phone.

Before receiving the first video stream sent by the terminal device, a triggering instruction may be received first to display a follow-up training interface on the display, where the follow-up training interface includes a first video playback window and a second video playback window, and the second video window is used for playback Fitness videos obtained from the server.

The triggering instruction may be triggered after the user performs an operation on the terminal device to trigger the display of the follow-up training interface.

In some embodiments, the triggering instruction may also be an instruction received by the display device from another control device to trigger display of the training interface.

S52. Obtain the scaling ratio according to the identification information, video playback parameters and the size of the first video stream.

Wherein, the video playback parameters include the size of the first video playback window used to play the first video stream and/or the preset video stream size.

In some embodiments, the scaling ratio may be obtained only based on the identification information, the size of the first video playback window, and the size of the first video stream, so that the size of the scaled first video stream is the same as that of the first video playback window. Adapt to the size of the window.

In some embodiments, the scaling ratio may be obtained only based on the identification information, the preset video stream size, and the size of the first video stream, thereby scaling the first video stream to a video stream of the preset video stream size.

In some embodiments, the scaling ratio can be obtained according to the identification information, the size of the first video playback window, the preset video stream size, and the size of the first video stream, so that the size of the scaled first video stream While adapting to the size of the first video playback window, it is more consistent with the aspect ratio of the original video. For example: the size of the first video playback window is 629×629. In order to make the aspect ratio of the projected video more consistent with the aspect ratio of the mobile phone in portrait mode, the default video stream size can be set to 450×629. Another example: the size of the first video playback window is 629×629. In order to make the aspect ratio of the screencast video more consistent with the aspect ratio of the mobile phone in landscape mode, the default video stream size can be set to 629×450. .

In some embodiments, the identification information includes: a first posture identification or a second posture identification.

When the identification information includes a first posture identification, the identification information is used to identify that the posture of the terminal device when recording the video corresponding to the first video stream is a portrait posture;

When the identification information includes a second posture identification, the identification information is used to identify that the posture of the terminal device when recording the video corresponding to the first video stream is a horizontal screen posture.

The video playback parameters corresponding to the first gesture identification and the video playback parameters corresponding to the second gesture identification are different.

The video playback parameters include the size of the first video playback window for playing the first video stream and/or the preset video stream size. In some embodiments of the present disclosure, different video playback parameters can be set for different gesture identifiers, That is to say, different gestures of the terminal device when recording the video corresponding to the first video stream may correspond to different sizes of the first video playback window and/or preset video stream sizes.

In some embodiments, when the identification information includes a first posture identification, it indicates that the posture of the terminal device when recording the video corresponding to the first video stream is a horizontal screen posture. In this case, the corresponding video playback parameters may be: The size of the first video stream is 1920×1080, and the size of the first video playback window is 626×629.

In some embodiments, when the identification information includes a second posture identification, it indicates that the posture of the terminal device when recording the video corresponding to the first video stream is a portrait posture. In this case, the corresponding video playback parameters may be: The size of the first video stream is 1080×1920, and the size of the first video playback window is 626×629.

S53. Scale the first video stream according to the scaling ratio to obtain the second video stream.

In some embodiments, the size of the first video stream is 1080×1920, and the scaling ratio is R, then the size of the second video stream is 1080*R×1920*R.

Through the scaling operation in step S53, the length or width of the first video stream can be adapted to the video playback parameters.

S54. Obtain a layout offset parameter according to the video playback parameter and the size of the second video stream.

Through the above step S53, the length or width of the second video stream can be adapted to the video playback parameters, but the other of the length and width of the second video stream needs to be adjusted to adjust the length and width of the second video stream. All are compatible with video playback parameters. In the embodiment of the present invention, the way to adapt the other one of the length and width of the second video stream to the video playback parameters is to adjust the second video stream between the length and the width when playing the video according to the second video stream. The width of the other one is offset in the direction. Therefore, in the embodiment of the present invention, it is necessary to first obtain the layout offset parameter according to the video playback parameters and the size of the second video stream, so that subsequent playback can be based on the second video stream. Layout offset during video playback.

S55: Offset the second video stream based on the layout offset parameter, and play the offset second video stream in the first video playback window.

In some embodiments, offsetting the second video stream based on the layout offset parameter may include: adjusting the origin of the coordinate system of the second video stream based on the layout parameter to offset the second video stream. offset. In other embodiments, offsetting the second video stream based on the layout offset parameter may also include: calculating each edge (top edge, bottom edge, left edge, etc.) of the second video stream based on layout parameter adjustment. edge and right edge) of the first video playback window, and then offset the second video stream according to the distance between the edges of the first video playback window corresponding to each edge of the second video stream.

The screen projection method provided by the embodiment of the present application, when receiving the first video stream generated by the video recording device of the terminal device and sent by the receiving terminal device, will first play the video according to the identification information carried by the first video stream. parameters and the size of the first video stream to obtain the scaling ratio, then scale the first video stream according to the scaling ratio to obtain the second video stream, and obtain the second video stream according to the video playback parameters and the size of the second video stream. The size obtains the layout offset parameter, and finally performs video playback in the first video playback window based on the second video stream, and offsets the second video stream based on the layout offset parameter. Because the identification information is used to identify the posture of the terminal device when recording the video corresponding to the first video stream, and the video playback parameters include the size of the first video playback window used to play the first video stream. and/or the video stream size is preset. Therefore, the embodiment of the present invention can zoom the first video stream according to the posture of the terminal device when recording the video corresponding to the first video stream, so that the obtained second video stream is in horizontally or vertically adapted to the video playback parameters; because the embodiment of the present invention will also obtain the layout offset parameter based on the video playback parameters and the size of the second video stream, and based on the layout during video playback The offset parameter offsets the second video stream, so this embodiment of the present invention can make the second video stream adapt to the video playback parameters in another direction. In summary, examples of the present invention can adapt the size of the screen projection video stream to the video playback window and/or the preset video stream size for playing the screen projection video stream in both horizontal and vertical directions.

As an expansion and refinement of the above embodiment, as shown in Figure 6, another screen projection method provided by an embodiment of the present invention includes:

S601. Receive the first video stream sent by the terminal device.

Wherein, the first video stream is a video stream obtained by video recording by a video recording device of the terminal device, and carries a message indicating that the posture of the terminal device when recording the video corresponding to the first video stream is vertical. Identification information of screen posture;

Referring to FIG. 7 , in FIG. 7 , the display device 71 is a television and the terminal device 72 is a mobile phone. The posture of the terminal device 72 when recording the video corresponding to the first video stream is a portrait posture.

S602. Obtain the first scaling ratio according to the width of the first video stream and the width of the first video playback window.

In the embodiment of the present invention, the width refers to the number of pixels included in the pixel row (horizontally), and the height refers to the number of pixels included in the pixel column (vertical).

Since the first video stream in the embodiment of the present invention is a video stream generated based on the video recording device of the terminal device, the size of the first video stream is the same as the resolution of the screen of the terminal device, for example: the screen of the terminal device The resolution is 1080×1920, and the size of the first video stream is 1080×1920.

In some embodiments, the first scaling ratio is a ratio of a width of the first video playback window to a width of the first video stream.

Assume: the first scaling ratio is R1, the size of the first video stream is W1×H1, and the size of the first video playback window is WL×HL, then there is:

R1＝WL/W1

In some embodiments, when the size of the first video stream is 1080×1920 and the size of the first video playback window is 626×629, the first scaling ratio R1=629/1080.

S603. Use the first scaling ratio to scale the first video stream to obtain the second video stream.

According to the above embodiment, the size of the first video stream is 1080×1920, the size of the first video playback window is 626×629, and the first scaling ratio R1=629/1080, then the first The size of the second video stream obtained by scaling the first video stream by the scaling ratio is 629×1118.

At this point, the width of the second video stream has adapted to the first video playback window.

S604. Obtain the first layout offset parameter according to the height of the second video stream and the height of the first video playback window.

In some embodiments, the first layout offset parameter is half of the difference between the height of the second video stream and the height of the first video playback window.

Suppose: the first layout offset parameter is CLIP ₁ , the size of the second video stream is W2×H2, and the size of the first video playback window is WL×HL, then there is:

CLIP ₁ ＝1/2(H2-HL)

In some embodiments, when the size of the second video stream is 629×1118 and the size of the first video playback window is 629×629, the first layout offset parameter CLIP ₁ =1/2 ( 1118-629).

S605. Offset the second video stream in the height direction based on the first layout offset parameter, and play the offset second video stream in the first video playback window.

Referring to FIG. 8 , the screen projection method provided in the above embodiment is explained by taking the size of the first video stream as 1080×1920 and the size of the first video playback window 800 as 629×629 as an example. . Referring to FIG. 8 , a first scaling ratio 629/1080 is obtained according to the width 1080 of the first video stream 81 and the width 629 of the first video playback window, and then the first video is scaled according to the first scaling ratio 629/1080. The stream 81 is zoomed to obtain the second video stream 82. The size of the second video stream 82 is 629×1118. Then, according to the height 1118 of the second video stream 82 and the height of the first video playback window 800, the first video stream 82 is obtained. Layout offset parameter 1/2 (1118-629) ≈ 244; finally, video playback is performed in the first video playback window 800 based on the second video stream 82, and the second video stream is offset upward by the second video stream. A layout offset parameter 244 to achieve the display effect shown in Figure 8.

As an expansion and refinement of the above embodiment, as shown in Figure 9, another screen projection method provided by an embodiment of the present invention includes:

S901. Receive the first video stream sent by the terminal device.

S902. Obtain the first scaling ratio according to the width of the first video stream and the width of the preset video stream size.

In some embodiments, the first scaling ratio is a ratio of a width in the preset video stream size to a width of the first video stream.

Assume: the first scaling ratio is R1, the size of the first video stream is W1×H1, and the preset video stream size is WS×HS, then there is:

R1＝WS/W1

In some embodiments, when the size of the first video stream is 1080×1920 and the preset video stream size is 450×629, the first scaling ratio R1=450/1080.

S903. Use the first scaling ratio to scale the first video stream to obtain the second video stream.

According to the above embodiment, the size of the first video stream is 1080×1920, the preset video stream size is 450×629, and the first scaling ratio R1=450/1080, then the first scaling ratio is used The size of the second video stream obtained by scaling the first video stream is 450×800.

At this point, the width of the second video stream is the same as the width of the preset video playback stream.

S904. Obtain the first layout offset parameter according to the height of the second video stream and the height of the first video playback window.

CLIP ₁ ＝1/2(H2-HL)

In some embodiments, when the size of the second video stream is 629×800 and the size of the first video playback window is 629×629, the first layout offset parameter CLIP ₁ =1/2 ( 800-629).

S905. Offset the second video stream in the height direction based on the first layout offset parameter, and play the offset second video stream in the first video playback window.

Referring to Figure 10, in Figure 10, the size of the first video stream is 1080×1920, the size of the first video playback window 1000 is 629×629, and the preset video stream size is 450×629. An example is provided to illustrate the screen projection method provided in the above embodiment. Referring to Figure 10, a first scaling ratio 450/1080 is obtained according to the width 1080 of the first video stream 101 and the width 450 of the first video playback window, and then the first video is scaled according to the first scaling ratio 450/1080. The stream 101 is zoomed to obtain the second video stream 102. The size of the second video stream 102 is 450×800. Then, according to the height 800 of the second video stream 102 and the height of the first video playback window 1000, the first video stream 102 is obtained. Layout offset parameter 1/2 (800-629) ≈ 85; finally, video playback is performed in the first video playback window 1000 based on the second video stream 102, and the second video stream is offset upward by the second video stream 102. A layout offset parameter of 85 is used to achieve the display effect shown in Figure 10.

The aspect ratio of the video finally displayed in the screen projection method shown in Figure 9 is closer to the aspect ratio of the screen of the terminal device in the portrait mode, so the projection screen of the terminal device can be better displayed.

As an expansion and refinement of the above embodiment, as shown in Figure 11, another screen projection method provided by an embodiment of the present invention includes:

S111. Receive the first video stream sent by the terminal device.

Wherein, the first video stream is a video stream obtained by video recording by a video recording device of the terminal device, and carries a message indicating that the posture of the terminal device when recording the video corresponding to the first video stream is horizontal. Identification information of screen posture;

Referring to FIG. 12 , in FIG. 12 , the display device 121 is a television and the terminal device 122 is a mobile phone. The posture of the terminal device 122 when recording the video corresponding to the first video stream is a horizontal screen posture.

S112. Obtain the second scaling ratio according to the height of the first video stream and the height of the first video playback window.

Similarly, the width here refers to the number of pixels contained in the pixel row (horizontally), and the height refers to the number of pixels contained in the pixel column (vertical).

In some embodiments, the first scaling ratio is a ratio of a height of the first video playback window to a height of the first video stream.

Assume that the second scaling ratio is R2, the size of the first video stream is W1×H1, and the size of the first video playback window is WL×HL, then there is:

R2＝HL/H1

In some embodiments, when the size of the first video stream is 1920×1080 and the size of the first video playback window is 626×629, the first scaling ratio R2=629/1080.

S113. Use the first scaling ratio to scale the first video stream to obtain the second video stream.

According to the above embodiment, the size of the first video stream is 1920×1080, the size of the first video playback window is 626×629, and the second scaling ratio R2=629/1080, then the first The size of the second video stream obtained by scaling the first video stream by the scaling ratio is 1118×629.

At this point, the height of the second video stream has adapted to the first video playback window.

S114. Obtain a second layout offset parameter according to the width of the second video stream and the width of the first video playback window.

In some embodiments, the second layout offset parameter is half of the difference between the width of the second video stream and the width of the first video playback window.

Assume: the second layout offset parameter is CLIP ₂ , the size of the second video stream is W2×H2, and the size of the first video playback window is WL×HL, then there is:

CLIP ₂ ＝1/2(W2-WL)

In some embodiments, when the size of the second video stream is 1118×629 and the size of the first video playback window is 629×629, the second layout offset parameter CLIP ₂ =1/2 ( 1118-629).

S115. Offset the second video stream in the width direction based on the first layout offset parameter, and play the offset second video stream in the first video playback window.

Referring to FIG. 13 , in FIG. 13 , the screen projection method provided by the above embodiment is explained by taking the size of the first video stream as 1920×1080 and the size of the first video playback window 1300 as 629×629 as an example. . Referring to Figure 13, a first scaling ratio 629/1080 is obtained according to the height 1080 of the first video stream 131 and the width 629 of the first video playback window, and then the first video is scaled according to the first scaling ratio 629/1080. The stream 131 is zoomed to obtain the second video stream 132. The size of the second video stream 132 is 1118×629. Then, according to the width 1118 of the second video stream 132 and the height of the first video playback window 800, the second video stream 132 is obtained. Layout offset parameters

Finally, video playback is performed in the first video playback window 1300 based on the second video stream 132, and the second video stream is offset to the left by the first layout offset parameter 244 to achieve the display shown in Figure 13 Effect.

As an expansion and refinement of the above embodiment, as shown in Figure 14, another screen projection method provided by an embodiment of the present invention includes:

S141. Receive the first video stream sent by the terminal device.

S142. Obtain the second scaling ratio according to the height of the first video stream and the height in the preset video stream size.

In some embodiments, the second scaling ratio is a ratio of a height in the preset video stream size to a height of the first video stream.

Assume: the second scaling ratio is R2, the size of the first video stream is W1×H1, and the preset video stream size is WS×HS, then there is:

R1＝HS/H1

In some embodiments, when the size of the first video stream is 1920×1080 and the preset video stream size is 450×629, the first scaling ratio R1=450/1080.

S143. Use the first scaling ratio to scale the first video stream to obtain the second video stream.

According to the above embodiment, the size of the first video stream is 1920×1080, the preset video stream size is 450×629, and the second scaling ratio R2=450/1080, then the second scaling ratio is used The size of the second video stream obtained by scaling the first video stream is 800×450.

At this point, the height of the second video stream is the same as the height of the preset video playback stream.

S144. Obtain a second layout offset parameter according to the width of the second video stream and the width of the first video playback window.

CLIP ₂ ＝1/2(W2-WL)

In some embodiments, when the size of the second video stream is 800×629 and the size of the first video playback window is 629×629, the second layout offset parameter CLIP ₂ =1/2 ( 800-629).

S145. Offset the second video stream in the width direction based on the first layout offset parameter, and play the offset second video stream in the first video playback window.

Referring to Figure 15, in Figure 15, the size of the first video stream is 1920×1080, the size of the first video playback window 1500 is 629×629, and the preset video stream size is 450×629. An example is provided to illustrate the screen projection method provided in the above embodiment. Referring to Figure 15, a first scaling ratio 450/1080 is obtained according to the height 1080 of the first video stream 151 and the height 450 of the first video playback window, and then the first video is scaled according to the first scaling ratio 450/1080. The stream 151 is zoomed to obtain the second video stream 152. The size of the second video stream 152 is 800×450. Then, according to the width 800 of the second video stream 152 and the width of the first video playback window 1500, the second video stream 152 is obtained. Layout offset parameter 1/2 (800-629) ≈ 85; finally, video playback is performed in the first video playback window 1500 based on the second video stream 152, and the second video stream is offset to the left by the second video stream. A layout offset parameter of 85 is used to achieve the display effect shown in Figure 15.

The aspect ratio of the video finally displayed in the screen projection method shown in Figure 16 is closer to the aspect ratio of the screen of the terminal device in the horizontal screen state, so the projection screen of the terminal device can be better displayed.

Referring to Figure 16, a screen projection method provided by another embodiment of the present invention includes:

S161. Receive the first video stream sent by the terminal device.

Wherein, the first video stream is a video stream obtained by video recording by a video recording device of the terminal device, and carries an identifier used to identify the posture of the terminal device when recording the video corresponding to the first video stream. information.

After any of the above-mentioned screen projection methods in Figures 5, 6, 9, 11, and 14, the subsequently received first video stream sent by the terminal device can also be processed and displayed in the manner shown in Figure 16 .

S162. Receive the trimming and playback instruction.

Wherein, the trimming and playback instruction carries the size of the selected area and the position information of the selected area, and is used to instruct video playback based on the video content of the first video stream in the selected area.

Referring to FIG. 17 , in FIG. 17 , the display device 171 is a television and the terminal device 172 is a mobile phone. The terminal device 172 determines the selected area 1700 in response to the user operation, and sends a trimming playback instruction to the display device 171 for instructing video playback based on the video content of the first video stream in the selected area.

In some embodiments, the size of the selected area and the selected area can be characterized by the position coordinates (x1, y1) of the upper right vertex of the selected area and the position coordinates (x2, y2) of the lower left vertex of the selected area. Location information for the region.

S163. In response to the trimming playback instruction, obtain a third zoom ratio according to the size of the selected area and the size of the first video playback window.

Wherein, the third scaling ratio is the smaller of the first ratio and the second ratio, and the first ratio is the ratio of the height of the first video playback window to the height of the selected area, so The second ratio is the ratio of the width of the first video playback window to the width of the selected area.

Assume: the size of the selected area is select_w×select_h, the size of the first video playback window is 629×629, and the third scaling ratio is R3, then there is:

R3＝min(629/select_w,692/select_h)

In some embodiments, when the size of the selected area is 265×360 and the size of the first video playback window is 629×629, the third scaling ratio R3=629/360.

In some embodiments, when the size of the selected area is 256×128 and the size of the first video playback window is 629×629, the third scaling ratio R3=629/256.

S164. Scale the first video stream using the third scaling ratio to obtain a third video stream.

In some embodiments, the size of the first video stream is 1920×1080, the size of the first video playback window is 629×629, and the size of the selected area is 480×270, then the size of the third The scaling ratio R3=629/480, and the size of the third video stream obtained by scaling the first video stream using the third scaling ratio is 2516×1415.

S165. Obtain a third layout offset parameter according to the third zoom ratio, the width of the first video playback window, the position information of the selected area, and the width of the selected area.

In some embodiments, the controller is specifically configured to calculate the third layout offset parameter through the following formula 1, where the formula 1 is:

CLIP ₃ =(X+select_w/2)*R3-WL/2

Wherein, CLIP ₃ is the third layout offset parameter, X is the distance from the left edge of the selected area to the left edge of the first video stream, and select_w/2 is the width of the selected area. , R3 is the third scaling ratio, and WL is the width of the first video playback window.

In some embodiments, when the size of the first video playback window is 629×629, the distance from the left edge of the selected area to the left edge of the first video stream is 1000, and the selected When the size of the area is 480×270, the third layout offset parameter CLIP ₃ =1310.

S166. Obtain a fourth layout offset parameter according to the third zoom ratio, the height of the first video playback window, the position information of the selected area, and the height of the selected area.

In some embodiments, the controller is specifically configured to calculate the fourth layout offset parameter through the following formula 2:

Assume that the fourth layout offset parameter is CLIP ₄ , the size of the third video stream is W3×H3, and the size of the first video playback window is WL×HL, then the second formula is:

CLIP ₄ =(Y+select_h/2)*R3-HL/2

Wherein, CLIP ₄ is the fourth layout offset parameter, Y is the distance from the top edge of the selected area to the top edge of the first video stream, select_h/2 is the height of the selected area , R3 is the third scaling ratio, and HL is the height of the first video playback window.

In some embodiments, when the size of the first video playback window is 629×629, the distance from the left edge of the selected area to the left edge of the first video stream is 400, and the selected When the size of the area is 480×270, the fourth layout offset parameter CLIP ₄ =386.

S167. Offset the third video stream in the width direction based on the third layout offset parameter, offset the third video stream in the height direction based on the fourth layout offset parameter, and playing the offset third video stream in the first video window.

In some embodiments, the second video stream is offset in the width direction based on the third layout offset parameter, and the third video stream is offset in the height direction based on the fourth layout offset parameter. Offsetting may include: adjusting a coordinate system origin of the third video stream based on a third layout offset parameter and the fourth layout offset parameter to offset the third video stream. In other embodiments, offsetting the second video stream based on the layout offset parameter may also include: calculating each edge (top edge, bottom edge, left edge, etc.) of the third video stream based on layout parameter adjustment. edge and right edge) of the first video playback window, and then offset the third video stream according to the distance between the edges of the first video playback window corresponding to each edge of the second video stream.

Referring to Figure 18, in Figure 18, the size of the first video stream is 1920×1080, the size of the first video playback window is 629×629, and the size of the selected area is 480×270. The distance from the left edge of the selected area to the left edge of the first video stream is 1000, and the distance from the top edge of the selected area to the top edge of the first video stream is 400. For example, The screen projection method provided by the above embodiment will be described. Referring to FIG. 18 , first, since 629/480<629/270, the third scaling ratio is determined to be 629/480. Secondly, the first video stream 181 is scaled according to the third scaling ratio to obtain the third video stream 182. The size of the third video stream is 2516×1415, and then the first video is played according to the third scaling ratio. The width of the window, the position information of the selected area, and the width of the selected area are used to obtain the third layout offset parameter 1310, and according to the third scaling ratio, the height of the first video playback window, the The position information of the selected area and the height of the selected area are obtained to obtain the fourth layout offset parameter 386. Finally, video playback is performed in the first video playback window 1800 based on the third video stream 182, and the The third video stream is offset to the left by the third layout offset parameter 1310 and upward by the fourth layout offset parameter 386 to achieve the display effect shown in FIG. 18 .

Based on any of the above embodiments, the screen casting method provided by the embodiment of the present invention further includes: receiving a second video stream sent by the server; and performing video playback in the second video playback window based on the second video stream.

In some embodiments, the above embodiments are described below by taking a video fitness scene as an example. As shown in FIG. 19 , the scene construction includes a fitness video server 191 , a display device 192 and a terminal device 193 . The display device 192 can obtain the fitness video from the fitness video server 191 and play it in the second video playback window 1922; the user can watch the fitness video played by the display device 192 and follow the fitness video for fitness training; the terminal device 193 can record through the video The device performs video recording to obtain the first video stream, and sends the first video stream to the display device 192. After the display device receives the first video stream sent by the terminal device 193, the screen projection method provided by the embodiment of the present invention is used to perform the first video stream. A video playback window 1921 is used to play the screencast video.

At present, when the TV is connected through the mobile phone as a camera device, in some scenarios, in order to capture the user, the general user needs to maintain a certain distance from the mobile phone. At this time, the user cannot see the mobile phone picture clearly. When the mobile phone is used as a camera device and connected to the TV, the battery of the mobile phone is exhausted, which is difficult for the user to notice, which will affect the basic functions of the mobile phone.

In some embodiments, in a scenario where a mobile phone is linked to a TV for fitness, or a scenario where a mobile phone is linked to a TV to dance, a certain distance needs to be maintained between the user and the mobile phone. At this time, the user cannot see the screen of the mobile phone clearly. , if the mobile phone is connected to the TV as a camera device to record the user practicing dance or fitness, the battery of the mobile phone will be exhausted and the user will not be able to detect it, which will affect the basic functions of using the mobile phone.

Considering the above problems, the display device according to some embodiments of the present disclosure further carries out the following design, and the screen projection method according to some embodiments of the present disclosure further includes a follow-up page control method. In addition, the present disclosure also provides a mobile terminal. After following a certain training video in response to a user trigger in the mobile terminal, the display can collect the follow-up video data through the mobile terminal, and determine the corresponding training video data and follow-up page identification according to the training video identification and follow-up page identification sent by the mobile terminal. The follow-up page data can also obtain the power data of the mobile terminal. In this way, the display can generate a follow-up page based on the follow-up video data, training video data and follow-up page data, and display the power of the mobile terminal in the follow-up page. data. Therefore, in a follow-up training scenario for fitness videos, dance videos, etc., when the user is at a certain distance from the mobile terminal, the power data of the mobile terminal is displayed on the follow-up training page of the display device. Since the user will Pay attention to the follow-up page on the monitor of the display device, so you will see the power data of the mobile terminal at the same time. In this scenario, you can discover the power consumption of the mobile terminal in time, charge it in time, and ensure the basic functions of the mobile terminal. .

In some embodiments, FIG. 1A is a schematic diagram of a scenario provided by an embodiment of the present disclosure. As shown in Figure 1, this scene includes: a terminal device 300, a display device 200 and a server 400.

The hardware configuration of the display device 200 according to some embodiments is shown in the block diagram shown in FIG. 3A.

In some embodiments, the terminal device 300 is a mobile terminal 300'. The mobile terminal 300' can install a software application with the display device 200, realize connection communication through a network communication protocol, and achieve the purpose of one-to-one control operation and data communication. The audio and video content displayed on the mobile terminal 300' can also be transmitted to the display device 200 to realize the synchronous display function. The display device 200 can also perform data communication with the server 400 through a variety of communication methods. The display device 200 may be allowed to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 can provide various content and interactions to the display device 200. In addition to providing broadcast reception and television functions, the display device 200 may also additionally provide an Internet television function that provides computer support functions.

The above-mentioned display device 200 can have a variety of implementation forms, for example, it can be a television, a television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. The above-mentioned mobile terminal 300' may also have various implementation forms, for example, a mobile phone, a tablet computer, a watch, etc. that can collect images and videos.

Figure 3B shows a hardware configuration block diagram of a mobile terminal 300' according to some embodiments. As shown in Figure 3B, the mobile terminal 300' includes: a controller 110, a detector 120, a communication interface 130, and a user input/output interface. (also called user interface) 140, display 150, as well as memory, power supply, etc. The communication interface 130 is used to communicate with the outside and includes at least one of a WIFI chip, a Bluetooth module, NFC or a replaceable module. The user interface 140 includes at least one of a microphone, a speaker, a touch pad, a sensor, a camera, or a replaceable module. Among them, the display 150 is used to display the user interface.

In some embodiments of the present disclosure, the detector 120 is configured to: collect follow-up video data;

The controller 110 is configured to obtain the power data of the mobile terminal; the communication interface 130 is configured to send the follow-up video data and the power data of the mobile terminal to a display device. In some embodiments of the present disclosure, the user interface 140 is configured to: receive the user's joint screen follow-up operation for the training video;

The detector 120 is configured to: in response to the joint-screen follow-up operation received by the user interface, collect follow-up video data;

The controller 110 is configured to: in response to the joint-screen follow-up training operation received by the user interface, determine the training video identification and follow-up page identification corresponding to the training video;

The controller 110 is also configured to: obtain the power data of the mobile terminal;

The communication interface 130 is configured to: send the training video identifier, the follow-up page identifier, the follow-up video data, and the power data of the mobile terminal to the display device.

In some embodiments of the present disclosure, the controller 110 is further configured to: cancel the display of the follow-up training page when it is determined that the power data of the mobile terminal indicates that the remaining power is less than or equal to the first power threshold; and controlling the communication interface to send a first notification message to the mobile terminal, where the first notification message is used to notify the mobile terminal to stop collecting the follow-up video data.

In some embodiments of the present disclosure, the controller 110 is further configured to: output first prompt information when it is determined that the power data indicates that the remaining power is less than or equal to the second power threshold, and the first prompt The information is used to prompt that the battery of the mobile terminal is too low;

Wherein, the method of outputting the first prompt information includes at least one of the following:

Control the display to display the first prompt information on the follow-up page;

Control the speaker to play the first prompt information.

In some embodiments of the present disclosure, the communication interface 130 is further configured to:

After the controller 110 outputs the first prompt information, it continues to receive the power data of the mobile terminal sent by the mobile terminal; and determines that the power data of the mobile terminal indicates that the remaining power is less than or equal to the second power. threshold, and is greater than or equal to the third power threshold, the first prompt information is no longer output, wherein the third power threshold is smaller than the second power threshold;

When it is determined that the power data of the mobile terminal indicates that the remaining power is less than or equal to the third power threshold, canceling the display of the follow-up page; and controlling the communication interface to send a first notification message to the mobile terminal, The first notification message is used to notify the mobile terminal to stop collecting the follow-up video data.

An output interface (display 260, and/or audio output interface 270) configured to output user interaction information;

The communication interface 220 is used to communicate with the server 400 or other devices.

In some embodiments, the display 260 is configured to display a user interface; the controller 250 is configured to obtain a training video identifier, a follow-up page identifier, follow-up video data, and power data of the mobile terminal. The video data is video data collected by the mobile terminal; determine the follow-up page data corresponding to the follow-up page identification, obtain the corresponding training video data from the server according to the training video identification; control the display 260 based on the follow-up page identification The training page data, the training video data and the following training video data display the following training page, the following training page includes a first window and a second window, and the training video data is displayed in the first window, The follow-up video data is displayed in the second window; the controller 250 is also configured to control the display to display the power data in the follow-up page.

In some embodiments, the communication interface 220 is configured to: receive the training video identifier, the follow-up page identifier, the follow-up video data and the power data of the mobile terminal sent by the mobile terminal, where the follow-up video data is the Video data collected by the mobile terminal; the controller 250 is configured to: determine the follow-up page data corresponding to the follow-up page identification, obtain the corresponding training video data from the server according to the training video identification; control the display based on the The following training page data, the training video data and the following training video data display the following training page, the following training page includes a first window and a second window, and the training video is displayed in the first window data, the following training video data is displayed in the second window;

The controller 250 is also configured to control the display to display the power data in the follow-up page.

In some embodiments of the present disclosure, the controller 250 is further configured to:

When it is determined that the power data of the mobile terminal indicates that the remaining power is less than or equal to the first power threshold, cancel the display of the follow-up page;

and controlling the communication interface to send a first notification message to the mobile terminal, where the first notification message is used to notify the mobile terminal to stop collecting the follow-up video data.

When it is determined that the power data indicates that the remaining power is less than or equal to the second power threshold, output first prompt information, where the first prompt information is used to prompt the mobile terminal that the power is too low;

Wherein, the method of outputting the first prompt information includes controlling the display to display the first prompt information on the follow-up page; and/or controlling the speaker to play the first prompt information.

In some embodiments of the present disclosure, the communication interface 220 is further configured to:

After the controller outputs the first prompt information, continue to receive the power data of the mobile terminal sent by the mobile terminal; and determine that the power data of the mobile terminal indicates that the remaining power is less than or equal to the second power threshold. , and is greater than or equal to the third power threshold, the first prompt information is no longer output, wherein the third power threshold is less than the second power threshold; when it is determined that the power data of the mobile terminal indicates remaining When the power is less than or equal to the third power threshold, cancel the display of the follow-up page; and control the communication interface to send a first notification message to the mobile terminal, where the first notification message is used to notify the mobile terminal Stop collecting the training video data.

The software configuration in the display device 200 or the mobile terminal 300' according to some embodiments is shown in FIG. 4A.

As shown in Figure 4B, a schematic diagram of software interaction for mobile terminals and display devices to implement follow-up page control methods according to some embodiments. The application framework, system library layer and kernel layer in the mobile terminal can be collectively referred to as the Android system layer, and the application program The framework, system library layer and kernel layer can be collectively referred to as the Android system layer, which can be represented as a mobile terminal APP in Figure 4B. The mobile terminal APP includes an APP application layer and a software development kit (Software Development Kit, SDK) layer. The SDK layer includes playback SDK and interactive SDK. The playback SDK encapsulates some common playback logic to decouple upper-layer applications from playback. The interactive SDK is responsible for the communication data transmission between mobile terminals and display devices, and supports mobile terminals to display devices. For control, the APP application layer is responsible for completing the user interface (UI) display and various interactive operations of the follow-up business on the mobile terminal, and reading the system power from the Android system layer to obtain the power data of the mobile terminal. The application framework, system library layer and kernel layer in the display device can be collectively referred to as the display device platform layer (an Android system customized based on the display device, which may be somewhat different from the Android system of the mobile terminal). In the embodiment of the present disclosure, the application The layer can run an application (Application, APP) that provides multimedia resources in the display device, which is represented as a display device APP in Figure 4B. The display device APP includes an APP application layer and a software development kit (Software Development Kit, SDK) layer, in which the SDK layer includes playback SDK and interactive SDK. Among them, the playback SDK encapsulates some common playback logic to decouple upper-layer applications from playback. The interactive SDK is responsible for the communication data transmission between the display device and the mobile terminal. APP The application layer is responsible for completing the follow-up business and displaying it on the user interface (UI) of the display device. It can be seen that the interactive SDK in the APP application layer of the mobile terminal can read the mobile terminal's power data, follow-up video data, and user-triggered training video identification and other information from the Android system layer and send them to the display device. Interactive SDK in the APP application layer, so that after receiving the data, the display device can display the follow-up page based on the data, and display the power of the mobile terminal on the follow-up page.

Some embodiments will be described below with reference to Figure 20. It can be understood that the steps involved in Figure 20 may include more steps or fewer steps during actual implementation, and the order between these steps may also be Different, whichever can implement the follow-up page control method provided according to some embodiments.

As shown in Figure 20, it is a schematic flowchart of a follow-up page control method provided by an embodiment of the present disclosure. The method may include but is not limited to the following steps:

S801. The mobile terminal receives the user's joint screen follow-up operation for the training video.

Among them, the above training video can be a fitness video or a dance video, etc.

In the embodiment of the present disclosure, the joint-screen follow-up operation refers to the operation of triggering the display device to enter the joint-screen follow-up mode. In the joint-screen follow-up mode, the selected content in the media asset platform (an application that provides multimedia resources) can be The media assets are played in the display device, and at the same time, the follow-up video data is obtained through the mobile terminal and played in the display device, where the display device is a device in the same WiFi network as the mobile terminal.

In some embodiments, an application program that provides multimedia resources may be installed on the mobile terminal. Correspondingly, the application program that provides multimedia resources may also be installed on the display device. The above training videos can be displayed on the application's pages.

The user can trigger the joint-screen follow-up operation for the training video through an application installed on the mobile terminal that provides multimedia resources. In some embodiments, a detailed page for joint-screen follow-up training for training videos can be set on the application. After the user clicks the "Start" control on this page, the mobile terminal receives the user's joint-screen follow-up training operation for the training video.

In the embodiment of the present disclosure, the mobile terminal and the display device can be connected through the same WiFi network.

S802. In response to the received joint-screen follow-up training operation, the mobile terminal determines the training video identifier and the follow-up training page identifier corresponding to the training video.

In this disclosed embodiment, each multimedia resource can be uniquely identified, and the above-mentioned training video identifier (Video ID) is used to uniquely identify the above-mentioned training video.

The above-mentioned follow-up page identifier is used to represent the follow-up page set based on the training video. The follow-up page identifier is a page number (Type Code). This page number can indicate the follow-up playback mode page, where the follow-up playback mode The page can be a full-screen follow-up playback mode page or a non-full-screen curtain playback mode page.

In some embodiments, in response to the joint-screen follow-up operation received by the user interface, the mobile terminal, in addition to determining the training video identification and follow-up page identification corresponding to the training video, can also determine the media asset identification (Media ID) corresponding to the training video. ). The media asset identifier may indicate media information corresponding to the training video, and the media information may include media information such as the title and poster of the training video.

In some embodiments, in response to the joint-screen follow-up operation received by the user interface, the mobile terminal can generate an operation instruction and send it to the display device. The operation instruction includes the above-mentioned training video identification and follow-up page identification.

In some embodiments, the above-mentioned operation instructions may include the above-mentioned training video identification, follow-up page identification and media asset identification. In some embodiments, the above-mentioned operation instructions may include the above-mentioned Video ID, Type Code, Media ID and execution operation (Action).

Among them, Action is used to represent the execution page jump and jump to the follow-up page corresponding to the training video.

In the case where the above operation instruction includes the above media asset identifier, after receiving the operation instruction, the display device may display some media resources of the training video while displaying the training video when displaying the follow-up page.

S803. The mobile terminal collects follow-up video data in response to the received joint-screen follow-up operation.

Among them, after the mobile terminal responds to the received continuous screen follow-up operation, it can call the camera to collect the follow-up video data in real time.

S804. The mobile terminal obtains the power data of the mobile terminal.

The mobile terminal can obtain the power data of the mobile terminal through the battery management module (BatteryManager) in the Android system layer.

The power data may include remaining power and/or remaining power percentage.

In some embodiments, the remaining power can be expressed as: 1000mA of power remaining, and the percentage of remaining power can be expressed as: 30% of power remaining.

S805. The mobile terminal sends the training video identifier, the follow-up page identifier, the follow-up video data, and the power data of the mobile terminal to the display device.

In the embodiment of the present disclosure, when the mobile terminal sends the training video identifier, the follow-up page identifier, the follow-up video data, and the power data of the mobile terminal to the display device, the mobile terminal can send it to the display device together through the same message, or it can also be sent separately. Sent to display device.

In the embodiment of the present disclosure, the terminal device and the display device establish a data transmission channel through the same WiFi network mentioned above, and realize the transmission of training video identification, follow-up page identification, follow-up video data, and power data of the mobile terminal through the data transmission channel. .

In some embodiments, the mobile terminal can collect the follow-up video data in real time, and establish a data transmission channel through the same WiFi network where the mobile terminal and the display device are located, and transmit the real-time collected follow-up video data to the display device in real time.

In some embodiments, the above-mentioned training video identification, follow-up page identification, and follow-up video data can be sent to the display device by the mobile terminal through operation instructions. After that, the mobile terminal can obtain the power data of the mobile terminal and send it to the display device. .

In some embodiments, after the mobile terminal receives the joint-screen follow-up operation, the terminal device can periodically obtain the power data of the mobile terminal and send the power data to the display device; accordingly, after receiving the joint-screen follow-up operation Before operation, the mobile terminal does not need to send power data.

In some embodiments, after the mobile terminal receives the joint-screen follow-up operation, the terminal device can periodically obtain the power data of the mobile terminal, and when the power data is less than or equal to the third power threshold, send the power data to the display device. The power data. Correspondingly, the mobile terminal may not send the power data before receiving the joint screen follow-up operation and/or when the power data is greater than the third power threshold.

Wherein, the third power threshold can be set according to actual needs.

In some embodiments, after the mobile terminal receives the joint-screen follow-up operation, the mobile terminal can periodically obtain the power data, and if the obtained power data changes compared with the previously obtained power data, The power data is sent to the mobile terminal; correspondingly, when the obtained power data has no change compared with the previously obtained power data, the power data does not need to be sent.

In some embodiments, the mobile terminal may first send the training video identifier, the follow-up page identifier, and the follow-up video data to the display device, so that the display device first displays the follow-up video identifier according to the training video identifier, the follow-up page identifier, and the follow-up video data. practice page, and notify the mobile terminal to successfully display the practice page. After the mobile terminal receives the notification, the terminal device can periodically obtain the power data of the mobile terminal and send the power data to the display device.

In some embodiments, the mobile terminal may first send the training video identifier, the follow-up page identifier, and the follow-up video data to the display device, so that the display device first displays the follow-up video identifier according to the training video identifier, the follow-up page identifier, and the follow-up video data. practice page, and notify the mobile terminal display device to successfully display the follow-up page. After the mobile terminal receives the notification, the terminal device can periodically obtain the power data of the mobile terminal. When the power data is less than or equal to the third power threshold Next, send the power data to the display device. Correspondingly, before the mobile terminal learns that the display device successfully displays the follow-up page, if the power data is greater than the third power threshold, the mobile terminal may not send the power data.

In some embodiments, the mobile terminal may first send the training video identifier, the follow-up page identifier, and the follow-up video data to the display device, so that the display device first displays the follow-up video identifier according to the training video identifier, the follow-up page identifier, and the follow-up video data. practice page, and notify the mobile terminal to successfully display the follow-up page. After the mobile terminal receives the notification, the terminal device can periodically obtain the power data of the mobile terminal, and compare the obtained power data with the previously obtained power data. If there is a change, the power data is sent to the mobile terminal; accordingly, if the obtained power data has no change compared with the previously obtained power data, the power data does not need to be sent.

The time period for the above-mentioned mobile terminal to obtain power data can be set according to the actual situation, and is not specifically limited in the embodiment of the present disclosure.

In some embodiments, the mobile terminal can also be used only as a camera. In this scenario, an application that provides multimedia resources can be installed on the mobile terminal. Correspondingly, the application program that provides multimedia resources can also be installed on the display device. The above training videos can be displayed on the application's pages. In this scenario, the display device can receive the user's joint-screen follow-up operation for the training video, and determine the training video identification, follow-up page identification, and media asset identification corresponding to the training video, based on the training video identification, follow-up page identification, and media asset identification. Generate a target QR code and display the target QR code. After the mobile terminal scans the target QR code, the mobile terminal can be triggered to obtain the follow-up video data through the camera, and send the follow-up video data to the display device. The display device The follow-up page data corresponding to the follow-up page identification can be determined, the corresponding training video data can be obtained from the server according to the training video identification, and the follow-up page can be displayed based on the follow-up page data, training video data and follow-up video data.

Among them, after the mobile terminal scans the target QR code through the application, the training video identification, follow-up page identification and media asset identification can be obtained. In the case that after the mobile terminal scans the target QR code, the TV exits the follow-up page due to user control or other reasons, the mobile terminal can send the obtained training video identification, follow-up page identification and media asset identification. to the display device, and sends the obtained follow-up video data to the display device, so that the display device can determine the follow-up page data corresponding to the follow-up page identification, obtain the corresponding training video data from the server according to the training video identification, and based on the follow-up Practice page data, training video data and follow-up video data show the follow-up page.

In some embodiments, the follow-up video data is continuous video streaming data sent in real time, so that the display device can display the local video picture on the local image window, and the power data can be sent periodically at predetermined time intervals, so as to reduce the mobile speed. device and display device loads. The preset time interval can be 3 minutes, 5 minutes, etc., which is much longer than the time between two frames of video in the follow-up video.

In some embodiments, the local video collected by the mobile terminal can be directly transmitted to the display device without being displayed on the mobile terminal.

In some embodiments, after establishing a video transmission connection, the mobile terminal stops the screen and allows an application that provides multimedia resources (such as a fitness APP) to work in the background, so as to realize local video collection and transmission while reducing the power consumption of the mobile terminal. .

In some embodiments, the transmission of local video collected by the mobile terminal (that is, the transmission of the above-mentioned follow-up video data) and the transmission of power data are two different transmission channels. In some embodiments, video transmission requires the video data to be According to the protocol encoding and transmission of the video format, the power data can be sent to the display device in the form of broadcast messages.

In some embodiments, the local video collected by the mobile terminal is transmitted by an application that provides multimedia resources (such as a fitness APP) through the mobile SDK, and the power data is transmitted by the power management application through the mobile SDK.

S806. The display device determines the follow-up page data corresponding to the follow-up page identification, and obtains the corresponding training video data from the server according to the training video identification.

The server can store various multimedia data corresponding to the multimedia resources in the application. The display device can determine the follow-up page data in the server based on the follow-up page identification, and obtain the corresponding training video data from the server based on the video identification.

In some embodiments, the display device may first determine the follow-up page data corresponding to the follow-up page identification, and then obtain the corresponding training video data from the server based on the training video identification. Alternatively, the display device may first obtain the corresponding training video data from the server based on the training video identification. training video data, and then determine the follow-up page data corresponding to the follow-up page identifier, or the display device can simultaneously perform the steps of determining the follow-up page data corresponding to the follow-up page identifier, and obtain the corresponding training from the server according to the training video identifier Video data is not limited in the embodiments of this disclosure. In some embodiments, determining the follow-up page data corresponding to the follow-up page identifier may include: different page identifiers are set correspondingly for different pages. After obtaining the follow-up page identifier (Type code), the display device may obtain the Type code. Follow-up page data corresponding to the set follow-up page. The follow-up page data may include the format information of the follow-up page (for example, color, page size, window size, window position, etc.), the follow-up page's Layout information, etc. The layout information of the follow-up page may indicate the source of the data displayed by the first window and the second window. In the embodiment of the present disclosure, it may indicate that the first window is used to display training video data obtained from the server, and the second window may be used to display the training video data obtained from the server. Used to display follow-up video data. After obtaining the follow-up page data, it can also be combined with the follow-up video data obtained from the mobile terminal.

In some embodiments, for the situation where the mobile terminal triggers the display device to display the follow-up page, when the mobile terminal sends the follow-up video data to the display device, the display device preferentially selects the follow-up video data sent by the mobile terminal to be displayed on the first page. In the second window, if the display device does not receive the follow-up video data sent by the mobile terminal, the camera of the display device can be turned on to obtain the follow-up video data and displayed in the second window.

In some embodiments, the situation where the mobile terminal triggers the display device to display the follow-up page may include: for example, the mobile terminal receives the user's joint-screen follow-up operation for the training video, and the mobile terminal collects the follow-up operation in response to the received joint-screen follow-up operation. Practice video data and send it to the display device; for another example, the display device can receive the user's joint screen follow-up operation for the training video, and determine the training video identification, follow-up page identification and media asset identification corresponding to the training video, based on the training video The target QR code is generated by the logo, the follow-up page logo and the media asset logo, and the target QR code is displayed. After the mobile terminal scans the target QR code, the mobile terminal can be triggered to obtain follow-up video data through the camera, and these tracking The training video data is sent to the display device.

In some embodiments, obtaining the corresponding training video data from the server according to the training video identification may include: different video identifications and different video data are correspondingly stored in the server. After obtaining the training video identification, the server may obtain the training video identification from the saved video identification. Determine whether the training video identifier exists, and if so, further obtain the training video data stored corresponding to the training video identifier.

In some embodiments, the corresponding training video data can also be obtained from the server by combining the media asset identifier and the training video identifier corresponding to the training video. The server also stores different media asset identifiers corresponding to the media asset information. The media asset information contains Including media information (for example, media information such as titles, posters, etc.) and multiple video data. After the server obtains the media asset identifier corresponding to the training video, it can first determine whether the media asset corresponding to the training video exists in the saved media asset identifier. If the identifier exists, the video data stored corresponding to the training video identifier is determined from multiple video data as the training video data based on the training video identifier.

S807. The display device displays the follow-up page based on the follow-up page data, training video data and follow-up video data.

The follow-up training page includes a first window and a second window. The first window displays training video data, and the second window displays follow-up video data.

The display device is based on the follow-up page data, training video data and follow-up video data. In the process of displaying the follow-up page, it can first draw the frame part of the follow-up page based on the follow-up page data, including drawing the frame part of the follow-up page. The first window and the second window, and then draw the training video data in the first window, and draw the follow-up video data in the second window, so that the complete follow-up page can be displayed. The training video data displayed in the first window of the follow-up page can be updated with the playback progress of the training video, and the content displayed in the second window of the follow-up page can be updated based on the follow-up video data obtained from the terminal device in real time. In other words, the display device needs to render the follow-up page based on the follow-up page data, training video data and follow-up video data in real time, and display the follow-up page.

As shown in Figure 21, it is a schematic diagram of a scene for displaying a follow-up page provided by an embodiment of the present disclosure. In Figure 21, the mobile terminal 92 collects the follow-up video data and sends it to the display device 91. The display device 91 displays the follow-up video data according to the mobile terminal. After the sent training video identification and follow-up page identification are determined, the follow-up page data and training video data can be displayed on the display screen of the display device 91. The follow-up page includes two windows, one of which is The training video data 911 is displayed, and the follow-up video data 912 is displayed in another window.

In the above embodiment, through the interaction between the mobile terminal and the display device, when the follow-up page is displayed, the user can watch the displayed training video data and perform follow-up exercises, and the mobile terminal will continue to collect the follow-up video data of the user's follow-up exercises. , and transmitted to the display device. The display device will display the follow-up video data and the training video data together on the display of the display device through different windows. In this way, the user can compare the training actions in the follow-up video data and the training video data, which is convenient for the user. Corrective action. In some embodiments, the video data may also be called a video stream, that is, the follow-up video data and the training video data are respectively called a follow-up video stream and a training video stream.

S808. The display device displays the power data on the follow-up training page.

In the embodiment of the present disclosure, during the process of displaying the follow-up training page, the obtained power data of the mobile terminal can also be displayed on the follow-up training page.

The power data can be displayed on the follow-up page in the form of icons and/or text.

As shown in Figure 22, it is a schematic diagram of displaying power data in a follow-up training page provided by an embodiment of the present disclosure. As shown in Figure 8, training video data 1001, follow-up video data 1002, and power data 1003 are displayed. The power data shown in FIG. 22 is displayed in the form of icons. In practice, it can also be displayed in other ways, which is not limited by the embodiment of the present disclosure.

In some embodiments, the fitness function demonstration video window and the local video window are displayed on the same first layer, and the power data is displayed on the second layer above the first layer.

In some embodiments, a power control is set on the second layer, and the display state of the power control changes with changes in the received power data.

In some embodiments, the power control on the second layer is generated by the display device according to the local configuration file of the display device, and is not an image uploaded to the screen by the mobile terminal. The configuration file corresponding to the power control can be set together with the configuration files of other windows, controls and other components in the follow-up interface, or it can be set separately from the configuration files in the follow-up interface.

The display device provided by the embodiment of the present disclosure includes: a display configured to display a user interface; a communication interface configured to receive a training video identifier and a follow-up page identifier sent by a mobile terminal; receive follow-up video data sent by the mobile terminal, The follow-up video data is video data collected by the mobile terminal; receives the power data of the mobile terminal sent by the mobile terminal; the controller is configured to determine the follow-up page data corresponding to the follow-up page identification, and obtain the corresponding follow-up page data from the server according to the training video identification Training video data; the control display displays the follow-up page based on the follow-up page data, training video data and follow-up video data. The follow-up page includes a first window and a second window. The first window displays the training video data, and the second window displays the training video data. Display the follow-up video data in the window; and control the display to display the power data on the follow-up page. Through this solution, after the mobile terminal responds to the user triggering a follow-up training video for a certain training video, the display can collect the follow-up video data through the mobile terminal, and determine the corresponding training video identification and follow-up page identification sent by the mobile terminal. The training video data and follow-up page data can also obtain the power data of the mobile terminal. In this way, the display can generate a follow-up page based on the follow-up video data, training video data and follow-up page data, and display it on the follow-up page. Power data of mobile terminal. Therefore, in a follow-up training scenario for fitness videos, dance videos, etc., when the user is at a certain distance from the mobile terminal, the power data of the mobile terminal is displayed on the follow-up training page of the display device. Since the user will Pay attention to the follow-up page on the monitor of the display device, so you will see the power data of the mobile terminal at the same time. In this scenario, you can discover the power consumption of the mobile terminal in time, charge it in time, and ensure the basic functions of the mobile terminal. .

In the embodiment of the present disclosure, in addition to displaying the power data in the follow-up training page, when the power data of the mobile terminal indicates that the remaining power is less than or equal to the set power threshold, prompt information can be output or the follow-up page can be controlled. , terminate the follow-up training and cancel the display of the follow-up training page.

When determining whether the remaining power of the mobile terminal is less than the set power threshold, the remaining power of the mobile terminal may be compared with the set power threshold to determine whether the remaining power of the mobile terminal is less than or equal to the set power threshold. , is still greater than the set power threshold.

In some embodiments, when the display device determines that the power data of the mobile terminal indicates that the remaining power is less than or equal to the first power threshold, the display device cancels display of the follow-up page and sends the first notification message to the mobile terminal. The first notification The message is used to notify the mobile terminal to stop collecting follow-up video data.

Before receiving the first notification message during the joint-screen follow-up training, the mobile terminal can call the camera of the mobile terminal to collect the follow-up video data in real time, and the preview interface of the user's follow-up training can be displayed in the above-mentioned application. The preview interface The real-time collected follow-up video data is displayed in the mobile terminal. The mobile terminal can send the follow-up video data to the terminal device. After the mobile terminal receives the above-mentioned first notification message, the above-mentioned application program of the mobile terminal can stop calling the camera of the mobile terminal. The training video data will no longer be collected, and the preview interface will be cancelled.

In some embodiments, when the power data of the mobile terminal indicates that the remaining power is less than or equal to the first power threshold, if it is detected within the first duration that there is a first character matching the first preset gesture in the follow-up video data. gesture, the mobile terminal stops collecting the follow-up video data and sends a second notification message to the display device; wherein the first preset gesture is a preset gesture used to indicate that it is allowed to stop follow-up practice, and the second notification message is used to notify The battery of the mobile terminal of the display device is too low, and the display of the follow-up training page is cancelled.

Wherein, when the power data of the mobile terminal indicates that the remaining power is less than or equal to the first power threshold, the user can be prompted for the remaining power of the current mobile terminal, and detect whether there is a third gesture matching the first preset gesture in the follow-up video data. A gesture. When the first gesture exists, it is determined that the user allows to stop the follow-up training. At this time, the mobile terminal is controlled to stop collecting the follow-up video data and sends a second notification message to the display device. Correspondingly, if the follow-up video data does not contain a gesture that matches the first preset gesture, the follow-up process will not be terminated.

In the above embodiment, when the power data of the mobile terminal indicates that the remaining power is less than or equal to the first power threshold, the follow-up process is not terminated directly, but the training process is collected by collecting whether the user performs the first gesture matching the first preset gesture. Gesture is used to indicate that it is allowed to stop following the practice. If the gesture is collected, the follow-up practice process will be terminated. In this way, the follow-up practice can be stopped according to the user's intention, which is more humane and improves the performance of human-computer interaction.

In some embodiments, when it is determined that the power data indicates that the remaining power is less than or equal to the second power threshold, first prompt information is output, and the first prompt information is used to prompt the mobile terminal that the power is too low.

The method of outputting the first prompt information includes controlling the display to display the first prompt information on the follow-up page, and/or controlling the speaker to play the first prompt information.

In some embodiments, the second power threshold is smaller than the first power threshold.

The above-mentioned first power threshold and second power threshold can be set according to actual needs, and are not limited in the embodiments of the present disclosure.

In some embodiments, when the power data of the mobile terminal indicates that the remaining power is less than or equal to the first power threshold, the collection of follow-up video data may be stopped and a second notification message may be sent to the display device. The second notification message may be The notification shows that the battery of the mobile terminal of the device is too low and the follow-up training page is cancelled.

As shown in Figure 23, it is a schematic flow chart of another follow-up page control method provided by an embodiment of the present disclosure. Combined with S808 shown in Figure 20, after the above S808, Figure 23 may also include but is not limited to the following steps:

S809. The mobile terminal sends the power data of the mobile terminal to the display device.

Among them, the mobile terminal can collect the power data of the mobile terminal in real time and send it to the display device. The mobile terminal can also collect the power data of the mobile terminal at preset intervals and send it to the display device.

In some embodiments, after collecting the power data of the mobile terminal, if the power data of the mobile terminal indicates that the remaining power is less than or equal to the second power threshold, the mobile terminal sends the power data of the mobile terminal to the display device.

When the power data of the mobile terminal indicates that the remaining power is less than or equal to the second power threshold, the mobile terminal sends the power data of the mobile terminal to the display device, which can avoid sending the power data to the display device multiple times when the remaining power of the mobile terminal is large. The display device sends the power data of the mobile terminal. The interaction is too frequent and has little practical effect, resulting in excessive power consumption and accelerating the power consumption of the mobile terminal.

S810. When the display device determines that the power data indicates that the remaining power is less than or equal to the second power threshold, output the first prompt information.

In some embodiments, as shown in Figure 24, a schematic diagram of displaying prompt information in the follow-up page is provided in an embodiment of the present disclosure. During the process of displaying the follow-up page, if the display device determines that the power data indicates the remaining power is less than or equal to the second power threshold, then the page shown in Figure 24 can be displayed. In addition to displaying the power data 1201 of the mobile terminal, the page also displays the first prompt information 1202. The first prompt information is displayed through text. "Mobile phone battery remaining 20%" prompts the user so that the user can know that the current mobile terminal battery is low and can charge in time or stop practicing.

In the embodiment of the present disclosure, in the scenario of joint-screen follow-up training, the user's focus is not on the terminal device (such as a mobile phone), but on the follow-up training page displayed on the display device (such as a television). For the user's focus in this scenario Different from other scenarios, in the embodiment of the present disclosure, the power data of the mobile terminal and the prompt information about the power are displayed on the follow-up page, so that the user can easily see the power data of the mobile terminal. Compared with The mobile terminal's own battery reminder solution. This display method of battery data and prompt information combines the characteristics of the actual scene, has the advantage of being easy to be discovered by users, and improves the performance of human-computer interaction.

Furthermore, in a joint-screen training scenario, there is a certain distance between the user and the terminal device. When the terminal device is a device with a small screen, such as a mobile phone or a phone watch, the display of power data and reminder information will be limited due to distance issues. It cannot be seen clearly by the user. Even if the user pays attention to the content displayed by the terminal device, it cannot have the effect of prompting the user. Therefore, in the case where the terminal device is a device with a small screen such as a mobile phone or a phone watch, the embodiment of this application The methods provided improve user experience and can improve human-computer interaction performance.

As shown in Figure 25, a schematic flow chart of another follow-up page control method provided by an embodiment of the present disclosure is shown in conjunction with Figure 23. After S810, the method shown in Figure 25 may also include the following steps:

S812. The mobile terminal sends the power data of the mobile terminal to the display device.

S813. When the display device determines that the power data of the mobile terminal indicates that the remaining power is less than or equal to the third power threshold, cancel the display of the follow-up page.

Wherein, the third power threshold is smaller than the second power threshold. The third power threshold may be the same as the first power threshold or may be different from the first power threshold. The third power threshold can be set according to actual conditions, and is not limited in this embodiment of the disclosure.

In some embodiments, the first power threshold and the third power threshold may be 5%, and the second power threshold may be 20%.

S814. The display device sends the first notification message to the mobile terminal.

The first notification message is used to notify the mobile terminal to stop collecting follow-up video data.

In the above embodiment, when the first prompt information is used to prompt that the battery is too low, and the user does not terminate the follow-up training, the remaining battery further reduces to the point where the remaining battery is less than or equal to the third battery threshold. In this case, you can forcefully cancel the display of the follow-up training page and terminate the follow-up training process to ensure that the mobile terminal will not run out of power during the follow-up training process.

In some embodiments, when the display device cancels the display of the follow-up training page and terminates the follow-up training, it may output a prompt interface for terminating the follow-up training and wait for the user to confirm whether to terminate the follow-up training within a certain period of time.

In some embodiments, as shown in Figure 26, a schematic diagram of prompting to terminate the follow-up practice in the follow-up practice page provided by the embodiment of the present disclosure can pop up a prompt window 1401 as shown in Figure 26 in the follow-up practice page. To remind the user that the battery of the mobile phone (i.e. mobile terminal) is low and the follow-up training has been suspended. Further users can exit the training page by selecting "OK, got it" on the control.

In some embodiments, assuming that the mobile terminal is a mobile phone and the display device is a television (television set), when interacting based on the software interaction diagram shown in Figure 4B, it involves the mobile phone application layer, mobile phone interaction SDK, TV application layer and As for the interaction between TV interactive SDKs, the follow-up page control method provided by the embodiment of the present disclosure will be described below based on the interaction between the mobile phone application layer, the mobile phone interactive SDK, the TV application layer, and the TV interactive SDK.

As shown in Figure 27, it is an interactive schematic diagram of a follow-up page control method provided by an embodiment of the present disclosure. The method may include the following steps:

S1. The mobile phone receives the user's click operation to enter the joint-screen fitness details page on the mobile APP and initiates the joint-screen fitness.

In this embodiment, the fitness and training scenario is taken as an example for explanation. The mobile application layer and mobile interactive SDK are set up for mobile APPs, while the TV application layer and TV interactive SDK are set up for TV APPs.

S2. After the mobile APP receives the user's operation, it notifies the mobile application layer (i.e. APP application layer). The mobile application layer calls the data sending (Send Data) interface provided by the interactive SDK and sends operation instructions to the mobile interactive SDK.

S3. The mobile phone interactive SDK transmits operation instructions to the TV interactive SDK.

Among them, the content of this operation instruction includes: the above-mentioned Media ID, Video Id, Type Code (page number-full-screen follow-up playback mode page), and Action (execution operation-page jump).

S4. TV interactive SDK sends operation instructions to the TV application layer.

S5. Obtain the media playback data of fitness training (i.e., training video data), open the fitness full-screen follow-up playback page (also called the fitness full-screen follow-up playback page), and display the follow-up page (also called the follow-up interface). Do follow-up exercises.

The content of the above-mentioned operation instruction also includes: an application identifier, and the application identifier is used to indicate the above-mentioned TV terminal APP.

After receiving the instruction content, the TV can determine which TV application needs to be launched based on the application identification, and determine based on the Type Code that the fitness full-screen follow-up playback page corresponding to the TV APP needs to be launched (i.e., the above-mentioned Follow the training page). And display the training video data in a window on the training playback page.

During this process, the mobile phone will also continuously send the follow-up video data collected in real time through the camera to the TV, so that after the TV receives the follow-up video data, it will be displayed on the follow-up page. The operation instruction and the training video data may be serially transmitted one after another. For example, after the mobile phone interactive SDK sends operation instructions to the TV interactive SDK, the mobile phone interactive SDK can send the follow-up video data collected by the camera in real time to the TV interactive SDK.

The mobile phone interactive SDK and the TV interactive SDK communicate through the same WiFi network they are connected to, and the command data is transmitted to the TV interactive SDK. After receiving the transmitted data, the TV interactive SDK notifies the upper-layer application through the callback function. After receiving the callback function, the TV application layer parses and obtains the identification (Type Code) of the follow-up page. According to the Type Code, it can call the system-side fitness details page interface (that is, the page interface corresponding to the follow-up page), and based on the Media ID , Video ID, obtain media asset playback data, launch the fitness full-screen follow-up playback page (i.e. follow-up page) and display it.

S6. The mobile phone APP (mobile application layer) obtains the current battery percentage through the Battery Manager of the Android system, and calls the interactive SDK interface to send data instructions for battery data. The content of the data instructions includes: mobile phone battery, and remaining battery percentage (data ).

Among them, the mobile phone interactive SDK is based on the network and transmits the data instructions to the TV interactive SDK.

S7, TV interactive SDK receives mobile phone power data transmitted by mobile phone interactive SDK.

S8, TV interactive SDK notifies the upper TV application layer of the mobile phone power data through the callback function.

Among them, the TV application layer can display based on the received power percentage data.

S9. When the battery percentage changes, the mobile application layer will transmit the latest battery data to the mobile phone interactive SDK.

S10 and mobile phone interactive SDK transmit the latest power data to TV interactive SDK.

S11. The TV interactive SDK sends the latest power data to the TV application layer.

The above-mentioned mobile phone APP will regularly monitor the power of the mobile phone (the current cycle can be once every minute). When the power percentage changes, the latest power data will be transmitted to the TV for the TV to refresh and display.

When the TV APP detects that the latest received data on mobile phone battery power is less than or equal to 20% (this threshold can be adjusted through the system), it will prompt and save the displayed power prompt in the memory for subsequent No more prompts.

In some embodiments, when the power of the mobile phone is less than 20%, the user charges the mobile phone and the power is restored to more than 20%. When the TV APP receives that the latest power transmitted by the mobile phone is greater than the current power and there is a low battery prompt flag in the memory, it clears the flag to 0. Once the battery falls below 20% again, the user will be prompted again.

S12. When the mobile phone APP detects that the current battery level is lower than 5%, the mobile phone application layer sends a stop command to the mobile phone interactive SDK.

When the mobile phone APP detects that the current battery level is lower than 5% (the threshold can be adjusted through the system), it automatically turns off the camera, exits the joint-screen fitness follow-up page, and sends a stop command to the TV. The content of the stop command includes: Action (execute operation-pause follow-up training).

S13. The mobile phone interactive SDK sends the stop command to the TV interactive SDK.

S14. The TV interactive SDK sends a stop command to the TV application layer.

Among them, when the TV APP receives the stop command, it suspends the current training and prompts the user that the mobile phone battery is low.

As shown in Figure 28, it is an interaction schematic diagram of another follow-up page control method provided by an embodiment of the present disclosure. The interaction process of this method is the same as the interaction process of steps S1 to S11 shown in Figure 27 above. The difference is that In the process shown in Figure 27 above, the user's follow-up stop (or pause) judgment logic occurs on the mobile phone, and the TV end is controlled to stop the follow-up practice by transmitting control instructions to the TV end. In the process shown in Figure 28, after step S11 , choose to implement the training stop (or pause) judgment logic on the TV side. That is, when the TV APP determines that the latest power data transmitted by the mobile phone reaches the power threshold for stopping follow-up training, the TV APP actively ends the current TV-side follow-up training and carries a notification to stop follow-up training in the return message through the interactive SDK.

With the development of terminal technology and Internet technology, multi-screen collaboration technology has been widely used. Multi-screen collaboration technology refers to the collaborative work of multiple screens, which includes projecting the screen content of one device to other devices for display. For example, the screen content of the mobile phone is projected to the computer for display, so that the user can watch the content displayed on the screen of the mobile phone on the computer, thus realizing the same screen display between the mobile phone and the computer.

In the related art, during the process of the first device projecting the screen to the second device, the second device displays the screen content of the first device. However, after the first device receives the communication message, the content displayed on the second device will be paused and the communication message will be accessed. However, in some usage scenarios, such as fitness scenarios, the user will follow the training and fitness video played by the second device. The first device collects the user's follow-up fitness video and projects it to the second device, and the second device displays the training and fitness video. and follow-up fitness videos. When the first device receives the communication message, it will interrupt the training and fitness videos and follow-up fitness videos played by the second device, which reduces the user's experience.

In order to solve the above technical problems or at least partially solve the above technical problems, the display device of some embodiments of the present disclosure further carries out the following design, and the screen projection method of some embodiments of the present disclosure further includes a communication message processing method.

Figure 1B shows a network architecture diagram of the screen projection system according to the screen projection method according to the embodiment of the present application. The network architecture diagram of the screen projection system includes a first device (also called a terminal device 300), a second device (also called a display device 200), a router 500, a server 400, etc. Among them: the first device can be used as the sending end in the network architecture of the screen projection system, and is used to send multimedia content and various instructions to the second device. In some embodiments, the first device includes, but is not limited to, a smartphone, a tablet, a game console, a laptop, or the like. The second device can be used as a display end in the network architecture of the screen projection system, used to receive multimedia content and various instructions from the first device, and synchronously play the received multimedia content to the display panel of its own device or the display of an associated device. in the panel. The second device may be a related device with a larger screen, such as a virtual reality device, a set-top box, a smart TV, a vehicle terminal, a large open-air display panel, etc. Of course, it may also be a device with a smaller screen, such as a smartphone, a tablet, a laptop, etc. relative devices.

The router 500 can be used as a gateway in the network architecture of the screen projection system to establish a local area network and provide Internet services. The router can provide a local connection between the first device and the second device.

The first device may discover the second device through the first network or the second network, and establish a network connection with the second device based on the first network or the second network. Through the established network connection, the first device can put the multimedia content that needs to be projected to the second device, and play it through the second device. The first network may be a local area network, and the local area network may be a wired LAN or a wireless LAN. As shown in Figure 1B, the wireless LAN may be a Wi-Fi network formed by a router 500. The second network may be a wide area network, such as the Internet. A wide area network can include physical links such as coaxial cable links, twisted pair cable links, fiber optic links, combinations thereof, etc. A wide area network may also include wireless links such as cellular links, satellite links, etc.

In some embodiments, the hardware configuration of the first device and the second device is shown in Figure 3A

Figure 29A is a schematic flow diagram of a communication message processing method according to some embodiments. Figure 29B is a schematic signal flow diagram of a communication message processing method according to some embodiments. With reference to Figures 29A and 29B, the communication message processing method includes:

S100. During the process of the first device projecting the screen to the second device, monitor the communication messages of the first device.

In S1001, the first device projects the screen to the second device; in S1002, the first device monitors communication messages.

In some embodiments, the first device includes an image acquisition module. In some embodiments, the image acquisition module may be a camera, and user video data is collected based on the image acquisition module of the first device.

In some embodiments, the second device plays a training video stream, for example, a fitness training video stream, a dance training video stream, the user follows the training video stream played by the second device, and the image collection module of the first device collects the user's follow-up training The video is streamed and displayed on the second device via screen casting.

Wherein, the distance between the first device and the second device is set to be less than a preset distance threshold, and the angle between the orientation of the image acquisition module of the first device and the orientation of the display screen of the second device is less than the preset angle threshold.

In some embodiments, when the user performs continuous body movements by following the training video stream played by the second device, for example, performing dance movements, gymnastics movements, etc., the image acquisition module of the first device is used to collect the user's training video stream. It can be understood that in order to keep the image acquisition module in a fixed position and prevent jitter, the first device can be installed on the fixed bracket, that is, the distance between the first device and the second device is less than the preset distance threshold, and the image acquisition module of the first device can The angle between the orientation and the orientation of the display screen of the second device is less than the preset angle threshold, ensuring that while the user can follow the training video stream played by the second device, the image acquisition module of the first device can collect the user's follow-up video stream. .

In some embodiments, during the process of the first device casting the screen to the second device, the second device can display the training video stream in the display window, and display the screencast content of the first device (i.e., follow-up video stream) in the screen casting window. ). For example, the screen projection window is used to display the screen content of the first device, that is, the display content in the screen projection window is the same as the content displayed on the screen of the first device.

It should be noted that screencasting usually includes push screencasting and mirroring screencasting. This disclosure is mainly used in mirror projection scenarios.

Among them, push screencasting refers to putting the multimedia resources of one device onto other devices for playback. Multimedia resources include audio and video resources. For example, putting videos on a mobile phone onto a smart TV for playback does not affect the user's use of the mobile phone, and the user can still operate the mobile phone. Mirroring refers to projecting the screen content of one device to other devices for display, that is, mirroring the entire screen of one device to display on other devices. For example, the screen content of a mobile phone is projected onto a projector for display. During this process, whatever content is displayed on the mobile phone will also be displayed on the projection window of the projector.

In addition, during the process of the first device casting the screen to the second device, the second device can also perform reverse control on the first device according to the screen casting window in the second device.

In some embodiments, when the first device projects the screen to the second device, in order to avoid the impact of the communication message received by the first device on the training video stream and follow-up video stream played by the second device, the first device In the process of casting the screen to the second device, the device first monitors the communication messages of the first device.

Communication refers to tools that can communicate. The content of messages can be voice, video, phone, text, etc.

The communication message indicates that the first device has received the communication message in the call request sent by the third device, where the call request includes: a cellular network call request (requires a SIM card, no application is required) or a non-cellular network call request (requires a SIM card) , application required), non-cellular network call requests may include: operator data network call requests, Internet call requests, LAN call requests, communication message types may be voice, video, phone, text, etc. (the third device sends a request to the first device There is an audio prompt message during the process of sending communication messages).

S200: After monitoring the communication message, obtain the status information of the follow-up video stream displayed by the second device.

In S2001, it is determined that the communication message to the device is monitored, and in S2002, status information of the follow-up video stream displayed by the second device is obtained.

Status information includes follow-up status and pause status.

The second device displays the follow-up video stream and the training video stream. The training video stream is the video stream played by the second device, and the follow-up video stream is the training video collected by the image acquisition module of the first device and the user follows the training played by the first device. The training video data corresponding to the video stream is then projected to the second device for display.

When the server of the second device monitors that the first device receives the communication message, the controller of the second device obtains the status information of the follow-up video stream displayed in the projection window of the second device, that is, based on the display of the projection window of the second device The status information of the follow-up video stream can be used to determine whether the user is following the training video stream played by the second device. If the status information of the follow-up video stream displayed by the second device is in the follow-up state, it can be determined that the user is following the training video stream played by the second device. If the status information of the training video stream played by the device is paused, it can be determined that the user has not followed the training video stream played by the second device.

In some embodiments, the status information of the follow-up video stream can be determined by continuously obtaining the image information of the follow-up video stream with preset frame values, and by comparing the image information corresponding to adjacent frame values. . In some embodiments, the image information corresponding to three consecutive frames of the follow-up video stream is obtained, and the image information corresponding to the first frame and the image information corresponding to the second frame are sequentially compared to see whether they are consistent, and the image information corresponding to the second frame is consistent with the image information corresponding to the second frame. Whether the image information corresponding to the three frames is consistent. When the image information corresponding to the first frame is consistent with the image information corresponding to the second frame, and the image information corresponding to the second frame is consistent with the image information corresponding to the third frame, then the follow-up video can be determined The stream is paused. When the image information corresponding to the first frame is inconsistent with the image information corresponding to the second frame, and/or the image information corresponding to the second frame is inconsistent with the image information corresponding to the third frame, it can be determined that the follow-up video stream is in the follow-up state.

S300. Send a control instruction to the second device according to the status information of the follow-up video stream displayed by the second device to control the second device to display notification content.

The notification content displayed by the second device corresponding to different control instructions is different.

When the first device projects the screen to the second device and the first device receives the communication message, the embodiment of the present disclosure obtains the status information of the follow-up video stream displayed by the second device and sends a control instruction to the second device to control The second device displays the notification content to prevent the first device from directly displaying the notification content corresponding to the communication message on the screen of the second device after receiving the communication message during the process of the first device projecting the screen to the second device, so that the communication message corresponds to The notification content interrupts the follow-up video stream and training video stream displayed on the second device, which reduces the user's experience.

The communication message processing method provided by the embodiment of the present disclosure monitors the communication messages of the first device during the process of casting the screen to the second device. After monitoring the communication message, the communication message is obtained by obtaining the follow-up message displayed by the second device. The status information of the video stream determines whether the user is following the training video stream played by the second device, and then sends a control instruction to the second device according to the obtained status information of the following video stream displayed by the second device to control the second device. Display notification content, that is, by obtaining the status information of the training video stream displayed by the second device, determine whether the user is following the training video stream played by the second device, and then control the control according to whether the user is following the training video stream played by the second device. The second device displays the notification content to avoid directly sending the notification content received by the first device to the second device when the user is following the training video stream played by the second device, pausing the training video stream played by the second device and following the training. Video streaming makes it impossible for users in the training state to respond in time, reducing the user experience.

Figure 30A is a schematic flow diagram of another communication message processing method according to some embodiments. Figure 30B is a schematic signal flow diagram of another communication message processing method according to some embodiments. With reference to Figure 30A and Figure 30B, the implementation of step S300 Methods include:

S301. When the status information of the follow-up video stream displayed by the second device is the follow-up state, send a first control instruction to the second device to control the second device to display text corresponding to the first notification content.

In S3011, when the status information of the follow-up video stream displayed by the second device is in the follow-up state, a first control instruction is sent to the second device, and in S3012, the second device is controlled to display text corresponding to the first notification content.

In some implementations, when the controller of the second device monitors that the first device receives the communication message and obtains the status information of the follow-up video stream displayed by the second device as the follow-up state, the controller of the second device Send a first control instruction to the second device to control the second device to display the text corresponding to the first notification content, that is, only the text corresponding to the notification content received by the first device is displayed, and the speaker does not play the audio corresponding to the notification content.

In some embodiments, if the first device opens a fitness application, the image collection module of the fitness application collects the user's follow-up video stream and projects it to the second device for display. When the first device projects the screen to the second device, During the process, the fitness application of the first device monitors the communication message sent by the third device to the first device. For example, the first device receives an incoming call from user A. At this time, the first device obtains the caller from the user's address book based on the incoming call. Information is sent to the second device by calling the middleware of the first device. After receiving the caller information sent by the middleware of the first device, the controller of the second device sends the text data corresponding to the caller information to the second device for processing. Screen projection display (the caller information is the notification content), that is, only the text of user A's call is displayed on the display screen of the second device, and the incoming call audio is not played, such as the incoming call music set by the user, to avoid the notification content of the first device The corresponding audio is played on the second device so that the audio corresponding to the notification content interrupts the follow-up video stream and the training video stream played by the second device.

S302. When the status information of the follow-up video stream displayed by the second device is in the paused state, send a second control instruction to the second device to control the second device to display the text corresponding to the second notification content and control the speaker to play the second notification content. The audio corresponding to the notification content.

In S3021, when the status information of the follow-up video stream displayed by the second device is in the paused state, a second control instruction is sent to the second device. In S3022, the second device is controlled to display the text corresponding to the second notification content and control the speaker to play the second notification. The audio corresponding to the content.

In some implementations, when the controller of the second device monitors that the first device receives the communication message and obtains the status information of the follow-up video stream displayed by the second device in the paused state, the controller of the second device sends The second control instruction is sent to the second device, the text corresponding to the second notification content, and the speaker is controlled to play the audio corresponding to the second notification content, that is, not only the text corresponding to the notification content received by the first device is displayed, but also the text corresponding to the notification content is played. audio.

In some embodiments, if the first device opens a fitness application, the image collection module of the fitness application collects the user's follow-up video stream and projects it to the second device for display. When the first device projects the screen to the second device, During the process, the fitness application of the first device monitors the incoming call from user A. At this time, the first device obtains the caller information from the user's address book according to the incoming call, calls the first device middleware and sends it to the second device. The second device After receiving the caller information sent by the middleware of the first device, the server sends the text data corresponding to the caller information to the second device for screen projection, and plays the incoming call music set by the first device, so that the user can pause the call. In the follow-up state, text data corresponding to the communication message is displayed and audio data is played through the second device.

Figure 31A is a schematic flow diagram of yet another communication message processing method according to some embodiments. Figure 31B is a schematic signal flow diagram of yet another communication message processing method according to some embodiments. The embodiment of the present disclosure is the implementation corresponding to Figure 30A Based on the example, as shown in Figure 31A and Figure 31B, the communication message processing method also includes:

S303. When the status information of the follow-up video stream displayed by the second device is in the follow-up state, in response to the pause follow-up instruction signal received by the second device, control the display to display the text corresponding to the notification content, and control the speaker to play the corresponding text audio.

In S3031, a pause follow-up instruction signal is received, in S3032, a second control instruction is sent to the second device, and in S3033, the display of the second device is controlled to display the text corresponding to the notification content and the speaker is controlled to play the audio corresponding to the notification content.

In some embodiments, when the communication message is monitored and the status information of the follow-up video stream displayed by the second device is the follow-up state, text corresponding to the notification content is displayed on the display to prompt the user that there is access to the communication message. On the basis of not affecting the user's follow-up training video stream, it can ensure that the user receives communication messages normally. The user can promptly receive prompt information that a communication message has been accessed based on the text corresponding to the notification content displayed on the display. At this time, the user can control the pause of the training video stream displayed by the second device through voice control or control device. For example, the user can use voice control to pause the training video stream displayed on the second device. Control the second device to pause the training video stream. The second device responds to the voice control signal output by the user and pauses the training video stream. At this time, the controller responds to the pause follow-up instruction information received by the second device, that is, the controller receives the second After receiving a signal indicating that the training video stream of the device is paused, the display is controlled to display the text corresponding to the notification content, and the speaker is controlled to play the audio corresponding to the notification content. That is, after receiving the instruction signal from the user to pause the training video stream played by the second device, through Control the display to display text corresponding to the notification content, and control the speaker to play the audio corresponding to the notification content, ensuring that users can receive notification content from a distance.

Figure 32A is a schematic flow diagram of another communication message processing method according to some embodiments. Figure 32B is a schematic signal flow diagram of another communication message processing method according to some embodiments. The embodiment of the present disclosure is the implementation corresponding to Figure 30A Based on the example, as shown in Figure 32A and Figure 32B, the communication message processing method also includes:

S304. When the status information of the follow-up video stream displayed by the second device is in the follow-up state, in response to the trigger operation received by the second device, send a first instruction instruction to the first device to instruct the first device to open the notification content corresponding Applications.

A trigger operation is received in S3040, a first control instruction is sent to the first device in S3041, and the first device opens an application corresponding to the notification content in S3042.

In some embodiments, the triggering operation includes an opening operation, and the triggering operation can be performed based on a control device or based on voice control.

When the status information of the follow-up video stream displayed by the second device is in the follow-up state, according to some embodiments, the user can trigger the notification content displayed by the second device through voice control or control device control. At this time, the controller responds to the user's The second device triggers the operation, sends the first instruction instruction to the first device, and opens the application/function corresponding to the notification content on the first device.

In some embodiments, if the first device monitors a communication message in application A, the communication message is that user A receives a video call from user B, and the status information of the follow-up video stream displayed on the second device is the follow-up status. At this time, the second device displays the text "User B's video call" corresponding to the notification content on the second device in the form of a message prompt. At this time, if the user triggers the text corresponding to the notification content "User B's video call" displayed on the second device. "Call", the controller responds to the text corresponding to the notification content triggered by the user on the second device, and sends a first instruction instruction to the first device to instruct the first device to open application A.

S305. Control the display to pause the training video stream and the follow-up video stream, and control the speaker to play the audio corresponding to the notification content.

In S3051, a second control instruction is sent to the second device; in S3052, the display of the second device pauses playing the training video stream and the follow-up video stream, and controls the speaker to play audio corresponding to the notification content.

When the status information of the follow-up video stream displayed by the second device is in the follow-up state, in some embodiments, the user can trigger the text corresponding to the notification content displayed by the second device through voice control or control device control. At this time, control The controller responds to the user's trigger operation on the second device, controls the display to pause and play the training video stream and the follow-up video stream, and controls the speaker to play the audio corresponding to the notification content.

In some embodiments, if the first device monitors a communication message in application A, the communication message is that user A receives a video call from user B, and the status information of the follow-up video stream displayed on the second device is the follow-up status. At this time, the second device displays the text "User B's video call" corresponding to the notification content on the second device in the form of a message prompt. At this time, if the user triggers the text corresponding to the notification content "User B's video call" displayed on the second device. "Call", the controller responds to the text corresponding to the notification content triggered by the user on the second device, controls the display to pause the playback of the training video stream and the follow-up video stream, and controls the speaker to play the audio corresponding to the notification content.

Figure 33A is a schematic flow diagram of yet another communication message processing method according to some embodiments. Figure 33B is a schematic signal flow diagram of yet another communication message processing method according to some embodiments. The embodiment of the present disclosure is the corresponding implementation in Figure 32A Based on the example, when the communication message processing method includes step S304, the method also includes:

S306. Monitor the communication status of the audio and video corresponding to the communication message of the first device.

S307. When the communication state of the audio and video corresponding to the communication message of the first device changes from the communication in progress state to the communication end state, establish a screen projection communication channel between the first device and the second device.

In S3071, the communication state of the audio and video corresponding to the communication message of the first device is monitored from the communication in progress state to the communication end state, and the communication channel is established again in S3072.

When the status information of the follow-up video stream displayed by the second device is in the follow-up state, the user can trigger the text corresponding to the notification content displayed by the second device on the second device, thereby causing the controller to send the first instruction instruction to the first instruction. device, causing the first device to open the application corresponding to the notification content. At this time, the user can answer the communication message on the application of the first device. Since the first device has answered the communication message, the first device exits and casts the screen to the second device. After the user answers the communication message on the first device, the controller monitors the status information of the audio and video corresponding to the communication message of the first device, and determines the time node when the first device ends the communication message. When the controller monitors the communication message corresponding to the first device, When the audio and video communication status changes from the communication state to the communication end state, the first device can send a screen casting request again, and the controller responds to the screen casting request sent by the first device and sends the screen casting request to the second device to achieve The screen projection channel between the first device and the second device is established again.

Figure 34A is a schematic flow diagram of yet another communication message processing method according to some embodiments. Figure 34B is a schematic signal flow diagram of yet another communication message processing method according to some embodiments. The embodiment of the present disclosure is the implementation corresponding to Figure 32A Based on the example, when the communication message processing method includes step S305, the method also includes:

S308. When the second device includes an application corresponding to the notification content, open the notification content through the application of the second device.

In S3081, it is determined that the second device includes an application corresponding to the notification content, and in S3082, the notification content is opened through the application of the second device.

S309. Monitor the communication status of the audio and video corresponding to the communication message.

S310. When the communication status of the audio and video corresponding to the communication message of the second device changes from the communication state to the communication end state, control the display to continue playing the training video stream and the follow-up video stream.

In S3101, it is determined that the communication status of the audio and video corresponding to the communication message of the second device changes from the communication state to the communication end state. In S3102, a second control instruction is sent to the second device. In S3103, the display is controlled to continue playing the training video stream and following the training. Video streaming.

When the status information of the follow-up video stream displayed on the second device is in the follow-up state, the user can trigger and open the text corresponding to the notification content displayed on the second device, thereby controlling the second device to pause the playback of the training video stream. and follow the video stream and play the audio corresponding to the notification content. At this time, when the second device includes an application corresponding to the notification content, the user can open the notification content through the second device to receive the communication message on the second device. Since the second device The device answered the communication message, and at this time, the follow-up video stream and training video stream played by the second device were paused. After the user answers the communication message on the second device, the controller monitors the status information of the audio and video corresponding to the communication message of the second device, and determines the time node when the second device ends the communication message. When the controller monitors the communication message corresponding to the second device, When the audio and video communication status changes from the communication status to the communication end status, the control display continues to play the training video stream and the follow-up video stream.

Figure 35A is a schematic flow diagram of yet another communication message processing method according to some embodiments. Figure 35B is a signal flow schematic diagram of yet another communication message processing method according to some embodiments. The embodiment of the present disclosure is the corresponding implementation in Figure 29A Based on the example, obtaining the status information of the follow-up video stream displayed by the second device includes:

S201. When it is detected that the prompt duration of the communication message received by the first device is greater than or equal to the preset threshold, obtain the status information of the follow-up video stream displayed by the second device.

In S2011, it is detected that the prompt duration of the communication message received by the first device is greater than or equal to the preset threshold, and in S2012, status information of the follow-up video stream displayed by the second device is obtained.

By monitoring the prompt duration of the communication message, when the prompt duration of the communication message is greater than or equal to the preset threshold, the status information of the follow-up video stream displayed by the second device is obtained, and when the prompt duration of the communication message is less than the preset threshold, it is ensured that The second device normally displays the follow-up video stream projected on the screen of the first device, and then determines that the communication message received by the first device is a useless communication message, and there is no need to access the communication message to the second device.

An embodiment of the present disclosure provides a display device. The display device includes: a display; and a controller configured to: monitor the communication messages of the first device during the process of projecting the screen from the first device to the second device; to the communication message, and obtain the status information of the follow-up video stream displayed by the second device. The status information includes the follow-up state and the pause state. The communication message represents the communication message sent by the third device to the first device; according to the follow-up video stream displayed by the second device, According to the status information of the video stream, a control instruction is sent to the second device to control the second device to display the notification content. The notification content displayed on the monitor corresponding to different control instructions is different.

The display device provided by the embodiment of the present disclosure monitors the communication messages of the first device during the process of the first device projecting the screen to the second device; and after monitoring the communication messages, obtains the content of the follow-up video stream displayed by the second device. Status information, the status information includes follow-up status and pause status; send control instructions to the second device according to the status information of the follow-up video stream displayed by the second device to control the second device to display notification content, and monitors corresponding to different control instructions The displayed notification content is different to ensure that when the status information of the follow-up video stream displayed by the second device is in the follow-up state, the communication message sent by the third device received by the first device will not affect the follow-up video stream displayed by the second device. and training video streams to improve users’ experience.

According to some embodiments, the controller is specifically configured to: when the status information of the follow-up video stream displayed by the second device is the follow-up state, send a first control instruction to the second device to control the second device to display the first notification. Text corresponding to the content; when the status information of the follow-up video stream displayed by the second device is in a paused state, send a second control instruction to the second device to control the second device to display the text corresponding to the second notification content and control the speaker Play the audio corresponding to the second notification content.

According to some embodiments, the controller is further configured to: when the status information of the follow-up video stream displayed by the second device is the follow-up state, in response to the pause follow-up instruction signal received by the second device, control the display to display the notification content corresponding to text, and controls the speaker to play the audio corresponding to the notification content.

The present disclosure also provides an electronic device, including: a processor, the processor is configured to execute a computer program stored in a memory, and when the computer program is executed by the processor, the steps of the above method embodiments are implemented.

FIG. 36 is a schematic structural diagram of an electronic device provided by the present disclosure. FIG. 36 shows a block diagram of an electronic device suitable for implementing some embodiments of the present invention. The electronic device shown in FIG. 36 is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

As shown in Figure 36, electronic device 800 is embodied in the form of a general computing device. The components of electronic device 800 may include, but are not limited to: one or more processors 810, system memory 820, and a bus 830 connecting different system components (including system memory 820 and processors).

Bus 830 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics accelerated port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect ( PCI) bus.

Electronic device 800 typically includes a variety of computer system readable non-volatile storage media. These media can be any media that can be accessed by electronic device 800, including volatile and nonvolatile media, removable and non-removable media.

System memory 820 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 840 and/or cache memory 850 . Electronic device 800 may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system 860 may be used to read and write to non-removable, non-volatile magnetic media (commonly referred to as "hard drives"). Disk drives may be provided for reading and writing from removable non-volatile disks (e.g., "floppy disks"), and for reading and writing from removable non-volatile optical disks (e.g., CD-ROMs, DVD-ROMs, or other optical media). CD-ROM drive. In these cases, each drive may be connected to bus 830 through one or more data media interfaces. System memory 820 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present invention.

A program/utility 880 having a set of (at least one) program modules 870, including but not limited to an operating system, one or more applications, other program modules, and programs, may be stored, for example, in system memory 820 Data, each of these examples or some combination may include an implementation of a network environment. Program modules 870 generally perform functions and/or methods in the described embodiments of the present invention.

The processor 810 executes at least one program among multiple programs stored in the system memory 820 to execute various functional applications and information processing, for example, to implement the method embodiments provided by the embodiments of the present invention.

The terminal device can perform screencasting operations to the display device. However, in the related technology, when the user follows the training video stream played by the display device, and the terminal device collects the user's follow-up video stream and projects the follow-up video stream to the display device, the terminal device will collect the user's training video stream. The follow-up video stream is rendered with special effects and then sent to the display device. However, the process of the terminal device rendering special effects on the user's follow-up video stream will inevitably lead to the inability to synchronize playback between the training video stream played by the display device and the follow-up video stream, which reduces the User experience.

The display device and the screen projection method of the embodiment of the present disclosure can be further implemented: after the display device receives the rendered follow-up video stream sent by the terminal device, it determines through comparison the training video stream played by the display device corresponding to the first playback moment. When the first training image does not match the first follow-up image action corresponding to the first playback moment in the rendered follow-up video stream, a first instruction signal is sent to the terminal device to instruct the terminal device to send the unrendered follow-up video. stream, that is, the display device directly receives the unrendered follow-up video stream sent by the terminal device. Therefore, it is possible to reduce the display delay between the training video stream displayed by the display device and the follow-up video stream due to the time-consuming operation of the terminal device rendering the follow-up video stream, and ensure that the follow-up video stream and training video displayed by the display device are Stream synchronization improves user experience.

The network architecture diagram of the screen projection system according to some implementations of the screen projection method of this application can be seen in Figure 1B.

The terminal device 300 and the display device 200 may also include a rendering module for providing audio or video special effects rendering services. The audio special effects may include audio reverberation, speed change, pitch change, limitation, echo, compression, gain control, equalization control, etc., and video special effects It can include beauty, makeup, face recognition, 2D stickers, 3D stickers, etc.

Figure 37A is a schematic flow diagram of a screen casting method according to some embodiments. Figure 37B is a schematic signal flow diagram of a screen casting method according to some embodiments. Figure 37C is an interaction schematic diagram according to some embodiments. Combined with Figure 37A, Figure 37B and Figure 37C, the screen projection methods include:

S1000. In response to the received start command, trigger the display to display the follow-up training interface.

Among them, the follow-up training interface is equipped with a display window and a local window, and the display window is used to display the training video stream.

The startup instruction may be a voice-based startup instruction, a startup instruction sent by a control device, or a startup instruction sent by an intelligent device.

In some embodiments, the control device may be a remote controller. The communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-distance communication methods to control the display device through wireless or wired methods. Users can control the display device by inputting user commands through buttons on the remote control, voice input, control panel input, etc.

In some embodiments, smart devices (such as mobile terminals, tablets, computers, laptops, etc.) can also be used to control the display device. For example, control a display device using an application running on a smart device.

In some embodiments, the display device may not use the above-mentioned smart device or control device to receive instructions, but may receive user control through touch or gestures.

In some embodiments, the display device can also be controlled in a manner other than the control device and the smart device. For example, the display device can directly receive the user's voice command control through a module configured inside the display device to obtain voice commands, or it can be controlled through The voice control device is set externally to the display device to receive the user's voice command control.

S2000: Obtain the training video stream corresponding to the follow-up training interface from the server, and receive the rendered follow-up video stream sent by the terminal device.

At S20001, the training video stream corresponding to the follow-up training interface is obtained from the server, and at S20002, the rendered follow-up video stream sent by the terminal device is received.

Among them, the rendered follow-up video stream is used to be displayed directly in the local window without being rendered by the display device.

The training video stream displayed on the display interface of the display is obtained from the server of the display device.

In some embodiments, the terminal device includes an image acquisition module. In some embodiments, the image acquisition module may be a camera, and the user video data is collected based on the image acquisition module of the terminal device.

As a specific implementation manner, the display device plays a training video stream, for example, a fitness training video stream, a dance training video stream, a training video stream played by the user through follow-up training on the display device, and the image acquisition module of the terminal device collects the user's follow-up training video flow.

Wherein, the distance between the terminal device and the display device is set to be less than a preset distance threshold, and the angle between the orientation of the image acquisition module of the terminal device and the orientation of the display screen of the display device is less than the preset angle threshold.

In the embodiment of the present disclosure, when the user performs continuous body movements through the training video stream played by the training display device, for example, performing dance movements, gymnastics movements, etc., the image acquisition module of the terminal device is used to collect the user's training video stream. It can be understood that in order to keep the image acquisition module in a fixed position and prevent jitter, the terminal device can be set on a fixed bracket, that is, the distance to the display device is less than the preset distance threshold, and the image acquisition module of the terminal device is oriented in the same direction as the display. The angle between the display orientations of the device is less than the preset angle threshold, ensuring that the user can follow the training video stream played by the display device and at the same time, the image acquisition module of the terminal device can collect the user's follow-up video stream.

In order to ensure the user's experience, in the existing technology, after collecting the follow-up video stream, the image acquisition module of the terminal device sends the collected follow-up video stream to the rendering module, and uses the rendering module to process the follow-up video stream. After the rendering operation is performed, it is sent to the display device to ensure that the follow-up video stream played on the display device is the rendered follow-up video stream.

S3000. When the action of the first training image corresponding to the first playback moment in the training video stream does not match the action of the first follow-up image corresponding to the first playback moment in the rendered follow-up video stream, search the training video stream for the first training image corresponding to the first playback moment. A second training image that matches the actions of the training image.

In S30001, it is determined that the first training image corresponding to the first playback moment in the training video stream does not match the action of the first follow-up image corresponding to the first playback moment in the rendered follow-up video stream. In S30002, search for the match in the training video stream. A second training image that matches the actions of the first training image.

The first playback time is greater than the second playback time corresponding to the second training image.

Since the terminal device needs to perform a rendering operation on the collected follow-up video stream, the rendering operation will take a certain amount of time. Therefore, there may be a problem that the rendered follow-up video stream received by the display device is out of sync with the training video stream, for example , at a certain playback moment, the training video stream is played to the tenth frame, but the rendered follow-up video stream is only played to the eighth frame, so that the rendered follow-up video stream and the training video stream viewed by the user on the display device are In different frames, the user cannot confirm whether the movements are consistent with the training video stream, which reduces the user experience.

Based on the problems existing in the existing technology, in the embodiment of the present disclosure, after receiving the rendered follow-up video stream sent by the terminal device, the first training image corresponding to the first playback moment in the training video stream played by the display device is first obtained. and the first follow-up image corresponding to the first playback moment in the rendered follow-up video stream, and then compare whether the actions of the first training image and the first follow-up image match. When the actions of the first training image and the first follow-up image do not match, When matching, it means that the training video stream played by the display device is out of sync with the follow-up video stream. The out-of-sync between the training video stream and the follow-up video stream may be caused by the terminal device needing to render the training video stream. At this time By searching for the second training image that matches the first following training image in the training video stream, that is, searching for the second training image that matches the first following training image before the first playback moment and the second training image in the training video stream. The second playback moment corresponding to the training image.

In some embodiments, searching for the second training image that matches the action of the first training image in the training video stream includes: in the training video stream, taking the first playback moment as the starting point, obtaining the first training video stream Play the candidate training image at a preset time before the time, and send the candidate training image and the first follow-up image to the server; obtain the server's comparison result of the candidate training image and the first follow-up image; compare it with the first follow-up image The candidate training images whose matching degree meets the preset threshold are determined as second training images.

In some embodiments, taking the first playback moment of the first training image of the training video stream played by the display device as the starting point, obtaining candidate training images at a preset time before the first playback moment of the training video stream, and comparing the first playback moment The matching degree between the candidate training image at the previous playback moment and the first follow-up image is determined when the matching degree between the candidate training image at the previous playback moment and the first follow-up image meets the preset threshold. The second training image is the candidate training image corresponding to the playback moment before the first playback moment; when the matching degree of the candidate training image at the playback moment before the first playback moment and the first follow-up image does not meet the preset threshold, Obtain the candidate training images corresponding to the first two moments of the first playback moment of the training video stream, and compare the matching degree of the candidate training images of the first two playback moments of the first playback moment with the first follow-up image. When the first playback moment When the matching degree between the candidate training images of the first two playback moments and the first follow-up training image meets the preset threshold, the second training image is determined to be the candidate training image corresponding to the first two playback moments of the first playback moment, and so on. , compare each candidate training image in sequence until the second training image is determined. If the comparison of each candidate training image ends and no candidate image is obtained whose matching degree with the first follow-up image meets the preset threshold, at this time, you can Determine that the second training image does not exist in the training video stream.

In some embodiments, the specific process of obtaining the server's comparison results of the candidate training image and the first follow-up image includes:

After acquiring the candidate training image and the first follow-up image sent by the controller of the display device, the server first extracts the first image feature point of the candidate training image and the second image feature point of the first follow-up image, and then based on image processing The algorithm compares the extracted first image feature points of the candidate training image and the second image feature point of the first follow-up image to determine the feature point comparison result.

S4000. When there is a second training image in the training video stream, send a first instruction signal to the terminal device to instruct the terminal device to send an unrendered follow-up video stream.

In S40001, it is determined that the second training image exists in the training video stream, in S40002, the first instruction signal is sent to the terminal device, and in S40003, the terminal device sends the unrendered follow-up video stream.

When the action of the first training image corresponding to the first playback moment in the training video stream does not match the action of the first follow-up image corresponding to the first playback moment in the rendered follow-up video stream, and there is a second training image in the training video stream , at this time, it means that the training video stream played by the display device is out of synchronization with the follow-up video stream. The first instruction signal is sent to the terminal device to instruct the terminal device to send the unrendered follow-up video stream. Since the follow-up video stream received by the display device The stream does not need to go through the rendering operation of the terminal device, that is, the display device directly receives the unrendered follow-up video stream from the terminal device. Therefore, it can reduce the training video stream displayed by the display device due to the time-consuming rendering operation of the follow-up video stream by the terminal device. The display delay between the training video stream and the training video stream.

The screen casting method provided by the embodiment of the present disclosure first responds to the received startup command, triggers the display to display the follow-up interface, then obtains the training video stream corresponding to the follow-up interface from the server, and receives the rendered follow-up sent by the terminal device. Video stream, when the first training image corresponding to the first playback moment in the training video stream does not match the first follow-up image action corresponding to the first playback moment in the rendered follow-up video stream, search the training video stream with The second training image matches the action of the first follow-up image. Finally, when the second training image exists in the training video stream, a first instruction signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up video stream, that is, After the display device receives the rendered follow-up video stream sent by the terminal device, it determines through comparison that the first training image corresponding to the first playback moment in the training video stream played by the display device and the rendered follow-up video stream corresponding to the first playback moment are compared. When the actions of the first follow-up image at a playback moment do not match, the first instruction signal is sent to the terminal device to instruct the terminal device to send the unrendered follow-up video stream, that is, the display device directly receives the unrendered follow-up video from the terminal device. stream, therefore, the display delay between the training video stream displayed by the display device and the follow-up video stream caused by the time-consuming rendering operation of the follow-up video stream by the terminal device can be reduced, ensuring that the follow-up video stream displayed by the display device is consistent with Training video streams are synchronized to improve user experience.

Figure 38A is a schematic flow diagram of another screen projection method according to some embodiments. Figure 38B is a signal flow diagram of another screen projection method according to some embodiments. The embodiments of the present disclosure are based on the above embodiments, combined with Figure 38A and Figure 38B, when a specific implementation of step S3000 includes:

S3001. Obtain the first training image of the training video stream at the first playback moment, and the first follow-up image corresponding to the first playback moment in the rendered follow-up video stream.

Since the follow-up video stream received by the display device is a rendered follow-up video stream, the first training image and the first follow-up image played by the terminal device may be different images. In this case, the training video stream is first obtained. The first training image at the first playback time and the first follow-up image corresponding to the first playback time in the rendered follow-up video stream.

S3002. When the action matching degree between the first follow-up image and the first training image is lower than the preset threshold, search for a second training image whose action matching degree with the first follow-up image meets the preset threshold from the training video stream.

After obtaining the first training image of the training video stream at the first playback moment and the first follow-up image corresponding to the first playback moment in the rendered follow-up video stream, the first follow-up image and the first training image are first determined The action matching degree of The reason why the follow-up video stream is out of sync is because the terminal device renders the follow-up video stream before sending it to the display device for display. Therefore, the playback progress of the follow-up video stream is slower than the playback progress of the training video stream. The training video stream of the follow-up image played by the stream has already been played. Therefore, the second training image matching the action of the first follow-up image can be found from the training video stream.

When the screen projection method includes step S3001 and step S3002, a specific implementation of step S4000 includes:

S4001. When the difference between the first playback time and the second playback time is greater than or equal to the preset difference, send a first instruction signal to the terminal device to instruct the terminal device to send an unrendered follow-up video stream.

In S40011, search the training video stream for a second training image whose action matching degree with the first training image meets the preset threshold. In S40012, determine that the difference between the first playback time and the second playback time is greater than or equal to the preset difference. value, in S40013, the controller sends a first instruction signal to the terminal device, and in S40014, the terminal device sends an unrendered follow-up video stream to the controller.

In some embodiments, after searching for the second training image whose action matching degree with the first training image meets the preset threshold from the training video stream, the second playback moment corresponding to the second training image can be determined, according to the first training The first playback time of the image and the second playback time of the second training image can determine the playback time at which the training video stream played by the display device is played, and the playback time at which the rendered follow-up video stream played by the display device is played. time, based on the relationship between the difference between the first playback time and the second playback time and the preset difference, the length of the delay time between the rendered follow-up video stream and the training video stream played by the display device is determined. When the first playback time When the difference from the second playback time is greater than or equal to the preset difference, at this time, it represents the delay time process between the rendered follow-up video stream played by the display device and the training video stream, resulting in the rendered The reason for the delay between the follow-up video stream and the training video stream may be that the terminal device needs to render the follow-up video stream collected by the image acquisition module, causing a delay in the terminal device sending the follow-up video stream data to the display device. , in order to ensure the synchronization of the follow-up video stream played by the display device and the training video stream, at this time, the controller of the display device sends a first instruction signal to the terminal device, instructing the terminal device to send an unrendered follow-up video stream, and the display device The controller directly receives the image acquisition module of the terminal device to collect the follow-up video stream, and does not perform rendering operations on the follow-up video stream to avoid the problem of out-of-synchronization between the training video stream played by the display device and the follow-up video stream due to the rendering operation.

When the screen casting method includes step S3001 and step S3002, other implementations of step S4000 include:

S4002. When the difference between the first playback time and the second playback time is less than the preset difference, do not send the first instruction signal to the terminal device.

In S40021, the terminal device sends the rendered follow-up video stream to the display device.

In some embodiments, as shown in FIGS. 38C and 38D , when the difference between the first playback moment and the second playback moment is less than the preset difference, it indicates that the rendered follow-up video stream played by the display device is different from the training video stream. The delay between video streams is within the acceptable range of the user, and the controller of the display device does not need to send the first instruction signal to the terminal device. At this time, the display device normally receives the rendered follow-up video stream sent by the terminal device, ensuring that the terminal device and The follow-up video streams displayed on the display device are all rendered follow-up video streams, which improves the user experience.

Figure 39A is a schematic flow diagram of another screen projection method according to some embodiments. Figure 39B is a signal flow diagram of another screen projection method according to some embodiments. The embodiments of the present disclosure are based on the above embodiments, combined with Figure 39A and Figure 39B, the screen projection method also includes:

S5000: Receive the unrendered follow-up video stream sent by the terminal device.

When the action of the first training image corresponding to the first playback moment in the training video stream does not match the action of the first follow-up image corresponding to the first playback moment in the rendered follow-up video stream, at this time, the rendered representation played by the display device There is a delay between the follow-up video stream and the training video stream. The reason for the delay between the rendered follow-up video stream played by the display device and the training video stream may be that the terminal device needs to collect the follow-up video stream collected by the image acquisition module. The video stream is rendered, causing a delay in the terminal device sending the follow-up video stream data to the display device. In order to ensure the synchronization of the follow-up video stream played by the display device and the training video stream, at this time, the controller of the display device sends the first The instruction signal is sent to the terminal device, instructing the terminal device to send an unrendered follow-up video stream, and the controller of the display device receives the unrendered follow-up video stream sent by the terminal device.

S6000. Determine the second training image in the unrendered training video stream that matches the action of the first training image of the training video stream.

Since in the initial state, the controller receives the rendered follow-up video stream sent by the terminal device, when the actions of the first training image of the training video stream displayed on the display and the first follow-up image of the follow-up video stream do not match, In order to ensure that the training video stream and the follow-up video stream displayed on the monitor are played synchronously, the controller first sends a first instruction signal to the terminal device, instructing the terminal device to send an unrendered follow-up video stream. However, because the rendered follow-up video stream played on the monitor The playback progress of the video stream is slower than the playback progress of the training video stream. After the controller obtains the unrendered follow-up video stream, if the controller directly renders the unrendered follow-up video stream from the second playback time and starts the second The follow-up video stream after the rendering operation is played during playback. At this time, the follow-up video stream and the training video stream played by the monitor are still out of sync. In the embodiment of the present disclosure, after receiving the follow-up video stream sent by the terminal device, first the second follow-up image that matches the first training image action of the training video stream is determined in the unrendered follow-up video stream.

S7000: Perform a rendering operation on the unrendered follow-up video stream and display the rendered follow-up video stream in the local window of the display. The follow-up video stream displayed in the local window is played from the playback time corresponding to the second follow-up image.

After determining the second training image that matches the action of the first training image of the training video stream from the unrendered training video stream, perform a rendering operation on the non-rendered training video stream and display the rendered image in the local window of the display. Follow-up video stream, and the follow-up video stream displayed in the local window is played from the playback time corresponding to the second follow-up image.

However, in order to ensure the user experience, the controller of the display device renders the unrendered follow-up video stream and then plays it on the display device. This avoids playing the unrendered follow-up video stream directly on the display device, which reduces user experience. The feeling is that the follow-up video stream played on the display device is the rendered follow-up video stream, and the unrendered follow-up video stream played on the receiving terminal device.

As an implementation manner, when there is a second training image in the training video stream, a first instruction signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up video stream. The first instruction signal is also used to indicate The end device pauses rendering operations.

In order to ensure that the follow-up video stream played by the display device is synchronized with the training video stream, the controller of the display device collects the follow-up video stream by directly receiving the image acquisition module of the terminal device to avoid the rendering operation causing the training video stream played by the display device to be inconsistent with the training video stream. To address the problem of out-of-synchronization of video streams, at this time, the first instruction signal that the controller of the display device can send is also used to instruct the terminal device to pause the rendering operation and reduce the power consumption of the terminal device.

An embodiment of the present disclosure also provides a display device. The display device includes: a display; and a controller configured to: in response to a received startup instruction, trigger the display to display a follow-up training interface, wherein the follow-up training interface is provided with a display window and The local window and the display window are used to display the training video stream; obtain the training video stream corresponding to the follow-up interface from the server, and receive the rendered follow-up video stream sent by the terminal device, where the rendered follow-up video stream is used without The display device renders directly in the local window; when the first training image corresponding to the first playback moment in the training video stream does not match the action of the first follow-up image corresponding to the first playback moment in the rendered follow-up video stream, in Search the training video stream for a second training image that matches the action of the first training image, where the first playback time is greater than the second playback time corresponding to the second training image; when there is a second training image in the training video stream, A first instruction signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up video stream.

As an implementation manner, the controller is further configured to: receive an unrendered follow-up video stream sent by the terminal device; determine in the unrendered follow-up video stream the first training image action that matches the training video stream. Second follow-up image; perform a rendering operation on the unrendered follow-up video stream and display the rendered follow-up video stream in the local window of the monitor. The follow-up video stream displayed in the local window starts from the playback time corresponding to the second follow-up image. Play.

As an implementation manner, the controller is further configured to not send the first instruction signal to the terminal device when the difference between the first playback time and the second playback time is less than a preset difference.

For convenience of explanation, the above description has been made in conjunction with specific implementation modes. However, the above discussion of some embodiments is not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Various modifications and variations are possible in light of the above teachings. The above embodiments are selected and described to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various modified embodiments suitable for specific use considerations.

Claims

A display device including:

monitor;

A controller, connected to the display, and the controller is configured to:

In response to the received startup instruction, a follow-up training interface is displayed on the display, wherein the follow-up training interface includes a first window and a second window, and the first window is used to play the training video stream obtained from the server, so The second window is used to play the training video stream received from the terminal device.
The display device of claim 1, wherein the controller is further configured to:

Receive a first video stream sent by a terminal device, wherein the first video stream is a video stream collected by a video recording device of the terminal device, and carries a corresponding message used to identify the terminal device for collecting the first video stream. The identification information of the posture in the video;

Obtain the scaling ratio according to the identification information, video playback parameters and the size of the first video stream, scale the first video stream according to the scaling ratio to obtain the second video stream, and obtain the second video stream according to the video playback parameters and The size of the second video stream obtains layout offset parameters; the video playback parameters include the size of the second window and/or the preset video stream size;

Offset the second video stream based on the layout offset parameter, and control the display to play the offset second video stream in the second window as a follow-up video stream.
The display device according to claim 2, wherein the identification information includes: a first posture identification or a second posture identification;

The first posture identifier is used to identify that the posture of the terminal device when recording the video corresponding to the first video stream is a portrait posture;

The second posture identifier is used to identify that the posture of the terminal device when recording the video corresponding to the first video stream is a horizontal screen posture;

Wherein, the video playback parameters corresponding to the first gesture identification and the video playback parameters corresponding to the second gesture identification are different.
The display device of claim 3, wherein in response to the identification information including the first gesture identification,

The controller scales the first video stream according to the scaling ratio to obtain a second video stream, and obtains layout offset parameters according to the video playback parameters and the size of the second video stream, including:

The controller obtains a first scaling ratio according to the width of the first video stream and a target width, where the target width is the width of the second window or the width in the preset video stream size; using the third Scale the first video stream by a scaling ratio to obtain the second video stream; obtain a first layout offset parameter according to the height of the second video stream and the height of the second window;

The controller offsets the second video stream based on the layout offset parameter including: the controller offsets the second video stream in a height direction based on the first layout offset parameter. .
The display device according to claim 4, wherein

The first scaling ratio is the ratio of the target width to the width of the first video stream;

The first layout offset parameter is half the difference between the height of the second video stream and the height of the second window.
The display device of claim 3, wherein in response to the identification information including the second gesture identification,

The controller obtains the scaling ratio according to the identification information, video playback parameters and the size of the first video stream, and scales the first video stream according to the scaling ratio to obtain the second video stream including:

The controller obtains a second scaling ratio based on the height of the first video stream and a target height, where the target height is the height of the second window or the height in the preset video stream size; using the Scale the first video stream with a second scaling ratio to obtain the second video stream;

The controller obtaining the layout offset parameter according to the video playback parameter and the size of the second video stream includes: the controller obtaining the layout offset parameter according to the width of the second video stream and the width of the second window. 2. Layout offset parameters;

The controller offsets the second video stream based on the layout offset parameter including:

The controller offsets the second video stream in a width direction based on the second layout offset parameter.
The display device according to claim 6, wherein

The second scaling ratio is the ratio of the target height to the height of the first video stream;

The second layout offset parameter is one-half of the difference between the width of the second video stream and the width of the second window.
The display device according to any one of claims 2-7, wherein the controller is further configured to offset the second video stream based on the layout offset parameter and control After the display plays the offset second video stream in the second window,

In response to the received trimming play instruction, a third zoom ratio is obtained according to the size of the selected area and the size of the second window, wherein the trimming play instruction carries the size of the selected area and the selected The location information of the area is used to indicate video playback based on the video content of the first video stream in the selected area, wherein the third scaling ratio is the smaller of the first ratio and the second ratio, The first ratio is the ratio of the height of the second window to the height of the selected area, and the second ratio is the ratio of the width of the second window to the width of the selected area;

Use the third zoom ratio to zoom the first video stream to obtain a third video stream; according to the third zoom ratio, the width of the second window, the position information of the selected area and the selected Obtain the third layout offset parameter based on the width of a certain area; obtain the fourth layout offset according to the third scaling ratio, the height of the second window, the position information of the selected area, and the height of the selected area. parameter;

Perform video playback in the second window based on the third video stream, offset the third video stream in the width direction based on the third layout offset parameter, and offset the third video stream based on the fourth layout offset parameter. The shift parameter offsets the third video stream in the height direction, and controls the display to play the offset third video stream in the second window.
The display device of claim 8, wherein the controller obtains a third zoom ratio, a width of the second window, position information of the selected area, and a width of the selected area. Three layout offset parameters include:

The controller calculates the third layout offset parameter through the following formula 1:

The first formula is: CLIP 3 = (X+select_w/2)*R3-WL/2;

The controller obtaining the fourth layout offset parameter according to the third scaling ratio, the height of the second window, the position information of the selected area, and the height of the selected area includes: the controller passes The third layout offset parameter is calculated using the following formula 2:

The second formula is: CLIP 4 = (Y+select_h/2)*R3-HL/2;

Wherein, CLIP 3 is the third layout offset parameter, CLIP 4 is the fourth layout offset parameter, and X is the distance from the left edge of the selected area to the left edge of the first video stream. , Y is the distance from the top edge of the selected area to the top edge of the first video stream, select_w/2 and select_h/2 are the width and height of the selected area respectively, and R3 is the Three scaling ratios, WL and HL are the width and height of the second window respectively.
The display device according to claim 1, wherein

The controller is also configured to: obtain a training video identifier, a follow-up page identifier, and follow-up video data as the follow-up video stream, where the follow-up video data is video data collected by the terminal device;

Determine the follow-up page data corresponding to the follow-up page identification, obtain the corresponding training video data as the training video stream from the server according to the training video identification; control the display based on the follow-up page data, the The training video data and the following training video data are displayed, and the following training page is displayed.
The display device according to claim 10, wherein the display device further comprises: a communication interface configured to: perform data communication with a terminal device;

The controller is also configured to:

Obtain the power data of the terminal device, and control the display to display the power data in the follow-up page;

In response to determining that the power data of the terminal device indicates that the remaining power is less than or equal to the first power threshold, canceling the display of the follow-up page; and

Control the communication interface to send a first notification message to the terminal device, where the first notification message is used to notify the terminal device to stop collecting the follow-up video data.
The display device according to claim 10, wherein the controller is further configured to:

Obtain the power data of the terminal device, and control the display to display the power data in the follow-up page;

In response to determining that the power data indicates that the remaining power is less than or equal to the second power threshold, output first prompt information, where the first prompt information is used to prompt the terminal device that the power is too low;

Wherein, the method of outputting the first prompt information includes at least one of the following:

Control the display to display the first prompt information on the follow-up page;

Control the speaker to play the first prompt information.
The display device according to claim 12, wherein the display device further comprises: a communication interface configured to: perform data communication with a terminal device;

The communication interface is further configured to: in response to the controller outputting the first prompt information, continue to receive the power data of the terminal device sent by the terminal device; and in response to determining the power of the terminal device The data indicates that the remaining power is less than or equal to the second power threshold and greater than or equal to the third power threshold, and the first prompt information is no longer output, wherein the third power threshold is less than the second power threshold;

In response to determining that the power data of the terminal device indicates that the remaining power is less than or equal to the third power threshold, canceling the display of the follow-up page; and

Control the communication interface to send a first notification message to the terminal device, where the first notification message is used to notify the terminal device to stop collecting the follow-up video data.
The display device according to claim 1, wherein

Controller, configured as:

In response to receiving the follow-up video stream from the terminal device, monitor the communication message of the terminal device;

In response to monitoring the communication message, obtain status information of the follow-up video stream displayed by the display device, where the status information includes a follow-up state and a pause state, and the communication message represents a third device sending to the terminal device communication messages;

According to the status information of the follow-up video stream displayed by the display device, a control instruction is sent to the display device to control the display device to display notification content. The notification content displayed by the display corresponding to different control instructions is different.
The display device according to claim 14, wherein

The controller is specifically configured as:

In response to the status information of the follow-up video stream displayed by the display device being the follow-up state, sending a first control instruction to the display device to control the display device to display text corresponding to the first notification content;

In response to the status information of the follow-up video stream displayed by the display device being in a paused state, sending a second control instruction to the display device to control the display device to display the text corresponding to the second notification content and control the speaker Play the audio corresponding to the second notification content.
The display device according to claim 14, wherein

The controller is also configured to:

Based on the status information of the follow-up video stream displayed by the display device being the follow-up state, in response to the pause follow-up instruction signal received by the display device, the display is controlled to display the text corresponding to the notification content, and the speaker is controlled to play the The audio corresponding to the notification content.
The display device according to claim 1, wherein

Obtain the training video stream corresponding to the follow-up interface from the server, and receive the rendered follow-up video stream sent by the terminal device, wherein the rendered follow-up video stream is used to directly display the training video stream on the third page without being rendered by the display device. Second window display;

In response to the mismatch between the first training image corresponding to the first playback moment in the training video stream and the first follow-up image action corresponding to the first playback moment in the rendered follow-up video stream, in the training Search the video stream for a second training image that matches the action of the first training image, wherein the first playback time is greater than the second playback time corresponding to the second training image;

In response to the presence of the second training image in the training video stream, a first instruction signal is sent to the terminal device to instruct the terminal device to send an unrendered follow-up video stream.
The display device according to claim 17, wherein

The controller is also configured to:

Receive an unrendered follow-up video stream sent by the terminal device;

Determine a second follow-up image in the non-rendered follow-up video stream that matches the action of the first training image of the training video stream;

After performing a rendering operation on the non-rendered follow-up video stream, the rendered follow-up video stream is displayed in the second window of the display. The follow-up video stream displayed in the second window is derived from the second follow-up image. Play at the corresponding playback time.
The display device according to claim 17, wherein

The controller is also configured to:

In response to the difference between the first playback time and the second playback time being less than the preset difference, the first instruction signal is not sent to the terminal device.
A method of screencasting, including:

In response to the received startup instruction, a follow-up training interface is displayed on the display of the display device, wherein the follow-up training interface includes a first window and a second window, and the first window is used to play the training video stream obtained from the server, The second window is used to play the training video stream received from the terminal device.
The method of claim 20, wherein:

Receive a first video stream sent by a terminal device, where the first video stream is a video stream collected by a video recording device of the terminal device, and carries a corresponding message used to identify the terminal device for recording the first video stream. The identification information of the posture in the video;

Obtain the scaling ratio according to the identification information, video playback parameters and the size of the first video stream, scale the first video stream according to the scaling ratio to obtain the second video stream, and obtain the second video stream according to the video playback parameters and The size of the second video stream obtains layout offset parameters; the video playback parameters include the size of the second window and/or the preset video stream size;

The second video stream is offset based on the layout offset parameter, and the offset second video stream is played in the second window as a follow-up video stream.
The method according to claim 21, wherein the identification information includes: a first posture identification or a second posture identification;

The first posture identifier is used to identify that the posture of the terminal device when recording the video corresponding to the first video stream is a portrait posture;

The second posture identifier is used to identify that the posture of the terminal device when recording the video corresponding to the first video stream is a horizontal screen posture;

Wherein, the video playback parameters corresponding to the first gesture identification and the video playback parameters corresponding to the second gesture identification are different.
The method of claim 20, wherein the method further includes:

Obtain a training video identifier, a follow-up page identifier, and follow-up video data as the follow-up video stream, where the follow-up video data is video data collected by the terminal device;

Determine the follow-up page data corresponding to the follow-up page identification, and obtain the corresponding training video data as the training video stream from the server according to the training video identification; based on the follow-up page data, the training video data and The following training video data displays the following training page.
The method of claim 23, wherein the method further includes:

Obtain the power data of the terminal device, and control the display to display the power data in the follow-up page;

In response to determining that the power data of the terminal device indicates that the remaining power is less than or equal to the first power threshold, canceling the display of the follow-up page; and

Control the communication interface to send a first notification message to the terminal device, where the first notification message is used to notify the terminal device to stop collecting the follow-up video data.
The method of claim 23, wherein the method further includes:

Obtain the power data of the terminal device, and control the display to display the power data in the follow-up page;

In response to determining that the power data indicates that the remaining power is less than or equal to the second power threshold, output first prompt information, where the first prompt information is used to prompt the terminal device that the power is too low;

Wherein, the method of outputting the first prompt information includes at least one of the following:

Control the display to display the first prompt information on the follow-up page;

Control the speaker to play the first prompt information.