WO2022089076A1

WO2022089076A1 - Display device, mobile terminal and picture synchronous scaling method

Info

Publication number: WO2022089076A1
Application number: PCT/CN2021/118592
Authority: WO
Inventors: 肖成创; 张传强; 林乐; 于颜梅; 程志; 郑鹏
Original assignee: 海信视像科技股份有限公司
Priority date: 2020-10-29
Filing date: 2021-09-15
Publication date: 2022-05-05

Abstract

Disclosed are a display device, a mobile terminal and a picture synchronous scaling method. The method can comprise receiving, by means of a mobile terminal, a scaling instruction input by a user, and scaling an original picture according to the scaling instruction. When a boundary of the original picture is beyond a boundary of a display area, a scaling instruction is sent to a display device. The scaling instruction comprises a display position and a scaling factor, and the display device can execute synchronous scaling on a target picture according to the display position and the scaling factor.

Description

Display device, mobile terminal, and picture synchronous scaling method

This application claims the priority of the Chinese patent application filed on October 29, 2020 with application number 202011179994.X; filed on February 09, 2021 with application number 202110175799.8, the entire contents of which are incorporated herein by reference Applying.

technical field

The present application relates to display device technologies, and in particular, to a display device, a mobile terminal, and a method for synchronous scaling of pictures.

Background technique

The smart TV can display the content in the mobile terminal by connecting to the mobile terminal such as the mobile phone. For example, a smart TV can establish a DLNA (Digital Living Network Alliance, Digital Living Network Alliance) and other means of image projection connection with a mobile terminal. After the connection relationship is established, the mobile terminal can send the displayed picture to the smart TV for display. And through the established connection relationship, operations such as zooming in, moving, etc. performed by the user on the mobile terminal can also be synchronously mapped to the smart TV, so as to realize the synchronous zooming of pictures.

SUMMARY OF THE INVENTION

Some embodiments of the present application provide a display device including a display, a communicator, and a controller. Wherein, the display is configured to display the target picture, the communicator is configured to establish a communication connection with the mobile terminal; the controller is configured to perform the following program steps: receiving a zoom instruction, the zoom instruction is caused by the mobile terminal The terminal inputs when the original picture boundary exceeds the display area boundary, and the scaling instruction includes a display position and a scaling ratio; the original picture is the picture displayed in the mobile terminal; in response to the scaling instruction, the target is moved according to the display position The position of the picture, the target picture is the picture displayed on the display; and the target picture is scaled according to the scaling ratio based on the display position.

Some embodiments of the present application also provide a mobile terminal, including a display unit, a communication unit, and a processing unit. The display unit is configured to display the original picture; the communication unit is configured to establish a communication connection with the display device; the processing unit is configured to execute the following program steps: receiving a zoom command input by a user; responding to the A zoom command, to perform zooming on the displayed original picture; if the boundary of the original picture exceeds the display area boundary of the display unit, calculate the display position and the zoom ratio; send a zoom command to the display device, the zoom command includes The display position and zoom ratio.

Some embodiments of the present application further provide a method for synchronously scaling a picture, including: a mobile terminal receiving a scaling command input by a user; the mobile terminal scaling the displayed original picture according to the scaling command; if the original picture exceeds the boundary of the original picture display area boundary, the mobile terminal sends a zoom instruction to the display device, the zoom instruction includes a display position and a zoom ratio; the display device moves the position of the target picture according to the display position; based on the display position, according to The scaling ratio performs scaling on the target picture.

Some embodiments of the present application provide a display device, including: a receiving module for receiving a transmission instruction; a driving module for determining a transmission mode in which the display device is located according to the transmission instruction; wherein the transmission mode is network transmission The main mode or the screen-casting transmission-based mode; the network transmission and the screen-casting transmission both perform data transmission through the same wireless communication module in the display device; The transmission mode determines the transmission duration adjustment instruction and sends the transmission duration adjustment instruction to the wireless communication firmware; the display screen is used to display the content of the network transmission and/or the screen projection transmission.

Description of drawings

FIG. 1 is a schematic diagram of an operation scenario between a display device and a control apparatus according to one or more embodiments of the present application;

2 is a block diagram of a hardware configuration of a display device according to one or more embodiments of the present application;

3 is a block diagram of a hardware configuration of a control device according to one or more embodiments of the present application;

4 is a schematic diagram of software configuration in a display device according to one or more embodiments of the present application;

5 is a schematic diagram showing an icon control interface of an application in a display device according to one or more embodiments of the present application;

6 is a schematic diagram of a mobile terminal according to one or more embodiments of the present application;

7 is a schematic diagram of interaction between a mobile terminal and a display device according to one or more embodiments of the present application;

8-9 are schematic diagrams of a synchronous zooming effect of pictures according to one or more embodiments of the present application;

FIG. 10 is a schematic diagram of display position calculation according to one or more embodiments of the present application;

FIG. 11 is a schematic diagram illustrating the effect of adjusting a picture position to a boundary by a display device according to one or more embodiments of the present application;

12 is a schematic diagram illustrating the effect of centering a picture position adjustment by a display device according to one or more embodiments of the present application;

13 is a schematic diagram of a synchronized zoom effect when a picture does not exceed the boundary of the display area according to one or more embodiments of the present application;

14 is a schematic diagram of a synchronized zoom effect when a picture exceeds the boundary of a display area according to one or more embodiments of the present application;

15 is a schematic diagram of a preset display area according to one or more embodiments of the present application;

16 is a schematic diagram illustrating the effect of zooming a target image in a preset display area according to one or more embodiments of the present application;

17 is a possible P2P device discovery process according to one or more embodiments of the present application;

18 is a process for adjusting a duration in a network transmission-based mode according to one or more embodiments of the present application;

19 is a process for adjusting a duration in a mode mainly based on screen projection transmission according to one or more embodiments of the present application;

FIG. 20 is a possible P2P device discovery process according to one or more embodiments of the present application.

Detailed ways

In order to make the objectives, implementations and advantages of the present application clearer, the exemplary embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the exemplary embodiments of the present application. Obviously, the exemplary embodiments described It is only a part of the embodiments of the present application, but not all of the embodiments.

Based on the exemplary embodiments described in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the appended claims of this application. Furthermore, although the disclosures in this application have been presented in terms of illustrative example or instances, it should be understood that various aspects of this disclosure may also constitute a complete embodiment in isolation. It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms are to be understood according to their ordinary and ordinary meanings.

FIG. 1 is a schematic diagram of an operation scenario between a display device and a control apparatus according to one or more embodiments of the present application. As shown in FIG. 1 , the user can operate the display apparatus 200 through the mobile terminal 100A and the control device. In some embodiments, the control device 100 may be a remote control 100B, and the communication between the remote control 100B and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-distance communication methods, etc., and the display device is controlled by wireless or other wired methods. 200. The user can control the display device 200 by inputting user instructions through keys on the remote controller 100B, voice input, control panel input, or the like. It should be noted that, the control apparatus 100 may communicate with the display device 200 in a direct wireless connection manner, and may also communicate in an indirect connection manner. In other embodiments, the control apparatus 100 may also access the same wireless network as the display device 200 through a wireless router, so as to establish an indirect connection and communication with the display device 200 through the wireless network. When sending the control command, the control device 100 may first send the control command data to the wireless router, and then forward the control command data to the display device 200 through the wireless router. In some embodiments, the mobile terminal 100A, tablet computer, computer, notebook computer, and other smart devices may also be used to control the display device 200 .

In some embodiments, the mobile terminal 100A can install a software application with the display device 200 to implement connection communication through a network communication protocol, so as to achieve the purpose of one-to-one control operation and data communication. As also shown in FIG. 1 , the display device 200 also performs data communication with the server 400 through various communication methods. The display device 200 may be allowed to communicate via local area network (LAN), wireless local area network (WLAN), and other networks. The server 400 may provide various contents and interactions to the display device 200 . The display device 200 may be a liquid crystal display, an OLED display, or a projection display device. The display device 200 may additionally provide a smart IPTV function that provides computer-supported functions, including but not limited to, IPTV, smart TV, Internet Protocol TV (IPTV), and the like, in addition to the broadcast receiving TV function.

FIG. 2 is a block diagram of a hardware configuration of a display device according to one or more embodiments of the present application. As shown in FIG. 2 , the control apparatus includes a controller 110 , a communication interface 130 , a user input/output interface 140 , a memory, and a power supply. The control device 100 can receive the user's input operation instruction, and convert the operation instruction into an instruction that the display device 200 can recognize and respond to, and play an intermediary role between the user and the display device 200 . The communication interface 130 is used for external communication, and includes at least one of a WIFI chip, a Bluetooth module, NFC or an alternative module. The user input/output interface 140 includes at least one of a microphone, a touchpad, a sensor, a key or an alternative module.

FIG. 3 is a block diagram of a hardware configuration of a control device according to one or more embodiments of the present application. As shown in FIG. 3, the display device 200 includes a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, and a controller. 250 , a display 260 , an audio output interface 270 , a memory, a power supply, and at least one of a user interface 280 . The controller includes a central processing unit, a video processing unit, an audio processing unit, a graphics processing unit, a RAM, a ROM, and a first interface to an nth interface for input/output. The display 260 may be at least one of a liquid crystal display, an OLED display, a touch display, and a projection display, and may also be a projection device and a projection screen. The tuner-demodulator 210 receives broadcast television signals through wired or wireless reception, and demodulates audio and video signals, such as EPG data signals, from a plurality of wireless or wired broadcast television signals. The detector 230 is used to collect external environment or external interaction signals. The controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

In some embodiments, the controller 250, through various software control programs stored on the memory, controls the operation of the display device and responds to user operations. The controller 250 controls the overall operation of the display apparatus 200 . A user may input a user command on a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user input interface recognizes the sound or gesture through a sensor to receive the user input command.

In some embodiments, a "user interface" is a medium interface for interaction and information exchange between an application program or an operating system and a user, which enables conversion between an internal form of information and a form acceptable to the user. The commonly used form of user interface is Graphical User Interface (GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be an icon, window, control and other interface elements displayed on the display screen of the electronic device, wherein the control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, Widgets, etc. at least one of the visual interface elements.

FIG. 4 is a schematic diagram of software configuration in a display device according to one or more embodiments of the present application. As shown in FIG. 4 , the system is divided into four layers, from top to bottom, the applications layer (referred to as “application layer”) ”), the Application Framework layer (referred to as the “framework layer”), the Android runtime (Android runtime) and the system library layer (referred to as the “system runtime layer”), and the kernel layer. The kernel layer contains at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power supply drive etc.

5 is a schematic diagram showing an icon control interface of an application in a display device according to one or more embodiments of the present application. As shown in FIG. 5 , the application layer includes at least one application that can display a corresponding icon control on the display, Such as: live TV application icon control, video on demand application icon control, media center application icon control, application center icon control, game application icon control, etc. Live TV app that can provide live TV from different sources. Video-on-demand application that can provide video from different storage sources. Unlike live TV applications, video-on-demand provides a display of video from certain storage sources. A media center application that can provide a variety of multimedia content playback applications. The application center can provide storage of various applications.

In some embodiments of the present application, the mobile terminal 100A refers to a terminal device capable of establishing a communication connection with the above-mentioned display device 200 . The mobile terminal 100A may serve as the control apparatus 100 of the display device 200 . The mobile terminal 100A includes, but is not limited to, a mobile phone, a computer, a tablet computer, a smart wearable device, and the like. For the convenience of description, some embodiments of the present application and the accompanying drawings in the specification take a mobile phone as an example for description. FIG. 6 is a schematic diagram of a mobile terminal according to one or more embodiments of the present application. As shown in FIG. 6 , functional devices such as a display unit 310, a communication unit 320, and a processing unit 330 may be built in the mobile terminal 100A to realize the foundation of the mobile terminal 100A. Features. Components such as memory, camera, sensor, audio circuit, and power supply can also be built-in to cooperate with other components to enable the mobile terminal to operate. That is, the display unit 310 may be used to receive input numerical or character information, and generate signal input related to user settings and function control of the mobile terminal 100A. The display unit 310 may also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal. The touch screen may be covered on the display screen, or the touch screen may be integrated with the display screen to realize the input and output functions of the mobile terminal 100A, which may be referred to as the touch screen screen after integration. In this application, the display unit 310 can display application programs and corresponding operation steps.

In some embodiments of the present application, the synchronous scaling of pictures means that the same picture is displayed on the mobile terminal 100A and the display device 200 respectively. When the user performs a scaling operation on the mobile terminal 100A, the picture displayed on the mobile terminal 100A will be scaled according to the scale. The operation is performed to zoom in or out, and at the same time, the picture displayed on the display device 200 is also zoomed in and out according to the zoom operation. In order to realize the synchronous scaling of the picture, before displaying the picture, it is necessary to establish a communication connection between the mobile terminal 100A and the display device 200, for example, connect the mobile terminal 100A and the display device 200 to the same wireless network through WiFi connection.

In some embodiments, FIG. 7 is a schematic diagram of interaction between a mobile terminal and a display device according to one or more embodiments of the present application. As shown in FIG. 7 , after a wireless connection is established, a user can perform an interactive operation on the mobile terminal 100A, The mobile terminal 100A is controlled to display pictures in synchronization with the display device 200 . For example, by performing an interactive operation, the mobile terminal 100A can simultaneously send a control instruction for synchronously displaying a picture and picture data to the display device 200 . The controller 250 of the display device 200 may first obtain the control instruction input by the mobile terminal 100A, and then, in response to the control instruction, receive picture data, thereby controlling the display 275 to display the picture data. Similarly, according to the layout of the UI interface of the display device 200, the display 275 may display picture data in a preset display area, and the preset display area may have an associated relationship with the display boundary in the mobile terminal 100A. The association relationship means that the state of the picture displayed on the display device 200 is compatible with the state of the picture displayed on the mobile terminal 100A. For example, when the mobile terminal 100A is placed horizontally, the preset display area of the display device 200 appears as a rectangular area whose width is greater than the height; when the mobile terminal 100A is placed vertically, the preset display area of the display device 200 appears as a rectangular area whose width is smaller than the height. rectangular area.

Obviously, in order to present a better display effect on the display device 200, the preset display area on the display device 200 can be set according to the screen characteristics of the display device 200, and automatically adapt to different forms of pictures. For example, the entire screen area can be used as the preset display area, and when displaying pictures of different ratios, the pictures are zoomed. In some embodiments, the display device 200 may obtain the picture data according to different ways. For example, the mobile terminal 100A may directly send the picture file to the display device 200, and the display device 200 will then display the received picture file. When the user inputs the zoom action instruction on the mobile terminal 100A, the mobile terminal 100A sends the zoom action instruction to the display device 200 after adjusting and converting the zoom action instruction. After receiving the zoom instruction, the display device 200 may execute the zoom instruction to also zoom the displayed picture.

However, for some display devices 200 and mobile terminals 100A whose screen shapes are quite different, the display forms of the same picture on the mobile terminal 100A and the display device 200 are different, so that the above-mentioned synchronous scaling method transmits the control instruction input by the user to the display device. After 200, the zoom command cannot be executed accurately. That is, the zoomed area on the mobile terminal 100A is likely to be inconsistent with the area displayed by the display device 200, resulting in a misalignment problem. For example, when the user displays a horizontal picture whose width is greater than its height in the vertical state of the mobile terminal 100A, in order to display the picture completely, the mobile terminal 100A will reduce the picture according to the width of the picture equal to the width of the display unit 310 of the mobile terminal 100A. Among them, the zoomed out picture is close to the middle area. However, since the width of the display device 200 is greater than the height, which is adapted to the shape of the picture, the display may not be reduced. When the user performs an operation of enlarging the upper left area on the mobile terminal 100A, since the zoomed-out image is close to the middle area, the user can input the enlargement action instruction in the area to the left and to the upper part of the center. At this time, the zoom command sent to the display device 200 is also in the area to the left and to the top in the middle, that is, the display device 200 enlarges the area to the left and above in the middle, not the upper left area, resulting in display dislocation.

In order to improve the problem of display misalignment, some embodiments of the present application implement synchronous scaling/moving between a mobile terminal and a display device when a picture is presented by using a picture synchronous scaling method. The method needs to establish a screen projection relationship between the display device and the mobile terminal, and display a target picture through the display, wherein the target picture is a picture with the same content as the original picture displayed on the display; the original picture boundary exceeds the display area of the mobile terminal. When the boundary is reached, the target image is scaled synchronously. Some embodiments of the present application provide a synchronous picture scaling method, and the picture synchronous scaling method can be applied to the display device 200 and the mobile terminal 100A. 8-9 are schematic diagrams of synchronous zooming effects of pictures according to one or more embodiments of the present application. As shown in FIGS. 8 and 9 , the method includes the following steps:

The mobile terminal 100A receives a zoom command input by the user. In some embodiments of the present application, after controlling the mobile terminal 100A to establish a communication connection with the display device 200, the user can control the display of pictures on the mobile terminal 100A and the display device 200 simultaneously through a picture projection or a related application. For ease of description, in some embodiments of the present application, the picture displayed on the mobile terminal 100A is called the original picture, and the picture displayed on the display device 200 is called the target picture. After the picture projection program is completed, the user can perform a zoom action on the mobile terminal 100A to control the original picture to be enlarged or reduced for display, that is, the mobile terminal 100A can receive a zoom command input by the user. Depending on the form of the mobile terminal 100A, the user may input the zoom command in different ways. For example, when the mobile terminal 100A is a mobile phone device with a touch screen, the user can input a zoom command by inputting a touch action of separating two fingers from each other in the picture display area on the touch screen of the mobile phone. A control UI may also be displayed on the mobile terminal 100A, and the user performs related operations through the controls in the UI interface, thereby realizing the zooming of the displayed original picture. For example, a scroll bar can be set on the right side of the picture display area. When the user slides the scroll bar up, the displayed picture is enlarged; when the user slides the scroll bar down, the displayed picture is reduced, thereby completing the input of the zoom command.

After receiving the zoom command input by the user, the mobile terminal 100A zooms the displayed original picture according to the zoom command. In some embodiments, in order to realize the zooming of the original image, the zooming command input by the user may include control parameters such as zooming position and zooming ratio. The zoom position may be determined according to the position where the user inputs the zoom command. For example, when the user inputs a zoom-in command by separating two fingers from each other, the zoom position may be the midpoint of the line connecting the two fingers. The zoom position can also be determined according to the display state of the picture. For example, when the user performs zooming using the scroll bar, the zooming can be performed according to the center position of the current display area, that is, the center position of the display is the zoom position. Obviously, in order to be able to zoom in and out of different positions of the picture, when displaying a larger size picture, move the picture position so that the area to be enlarged is located in the center of the display area, and then drag the scroll bar to zoom in or out. The zoom ratio can also be determined according to the zoom command input by the user. For example, the zoom ratio may have a relationship with the sliding distance of the user inputting the action of separating two fingers from each other, that is, as the distance between the two fingers increases, the original image is gradually enlarged, and the further the two fingers move, the higher the magnification. The scaling can also be determined based on how far the user drags the scroll bar to move. That is, the corresponding relationship between the zoom ratio and the sliding distance can be calibrated on the scroll bar first, and when the user moves the scroll bar to any position, the picture is zoomed to a corresponding multiple.

Therefore, after receiving the zoom command, the mobile terminal 100A may parse the zoom command to determine the zoom position and zoom ratio specified in the zoom command. Then, according to the zoom position and zoom ratio, perform a zoom operation on the original image to enlarge or reduce the original image. It should be noted that, in order to present a better zoom-in and zoom-out effect, in practical applications, multiple samplings can be used to determine the zoom position of each sampling process and the zoom ratio between two adjacent sampling processes, and perform the zoom operation. The displayed picture can be gradually enlarged or reduced according to the input process of the zoom command, so as to realize the effect of gradually enlarging or reducing the picture following the user's input operation action.

The screen size of the mobile terminal 100A is generally small. When the original picture is displayed on the mobile terminal 100A, it is usually in a reduced state. After the screen is projected to the display device 200, the large screen size of the display 275 can be adjusted according to the original picture size. show. Therefore, when the original picture is displayed on the mobile terminal 100A, there is a certain interval between a part of the boundary of the display area and the picture boundary due to the reduced display of the picture. And as the enlarging process progresses, the picture boundary gradually approaches the display area boundary. When the picture boundary reaches the display area boundary, it means that the current mobile terminal 100A can realize the maximum complete display state of the original picture in this direction, which is usually the original picture size.

According to the zooming method of the image, the direction of the border of the display area that the image gradually approaches with the zooming process is different. For example, when the mobile terminal 100A is placed vertically, that is, the width of the display screen is smaller than the height, when displaying a widescreen image, the mobile terminal 100A will scale the image according to the width of the image, so that the width of the image is equal to the width of the display area. At this time, there are gaps between the upper and lower borders of the display area and the top and bottom edges of the picture. In the same way, when the mobile terminal 100A is placed horizontally and a portrait screen picture is displayed, the mobile terminal 100A will be reduced in the form that the height of the picture is equal to the height of the display area. There are gaps between.

For ease of description, in some embodiments of the present application, the boundary of the display area that can be gradually approached is referred to as the zoom boundary. When the mobile terminal 100A executes the zoom-in command, due to the zoom-in effect on the picture, the picture border will gradually approach the zoom border. After the boundaries of the two are overlapped, if the enlargement command is continued to be input, the mobile terminal 100A will continue to enlarge the picture beyond the original size of the picture, so that the picture cannot be fully displayed. Correspondingly, the displayed picture also needs to be synchronously enlarged on the display device 200 . That is, in some embodiments of the present application, with the input of a zoom instruction by the user, if the border of the original picture exceeds the border of the display area, the mobile terminal 100A sends a zoom instruction to the display device. When the original picture boundary exceeds the scaling boundary, the mobile terminal 100A can send a scaling instruction to the display device 200, so that the display device 200 can perform scaling on the displayed target picture according to the scaling instruction, wherein the scaling instruction includes a display position and a scaling ratio. In some embodiments of the present application, the display position may be represented by the center point of the display area of the mobile terminal 100A, and the pixel value position of the enlarged image corresponds to the pixel point on the original image. The zoom ratio can be based on the zoom in/out factor of the image that reaches the display boundary.

Through the above-mentioned sending method of the zoom command, when the zooming boundary of the picture displayed by the mobile terminal 100A is reached, the zoom command is not sent, and only the original image is zoomed in the mobile terminal 100A, so that on the premise of satisfying the display size of the original image, there is no need to send the zoom command. The target picture is zoomed; and after the picture on the mobile terminal 100A fills the display area in the direction of the zoom boundary, the zooming instruction starts to be sent to the display device 200 to realize synchronous zooming.

After receiving the zoom instruction, the display device 200 may move the position of the target picture according to the display position. Because the zoom position specified when the user inputs the zoom command may not correspond completely due to the difference in screen size between the mobile terminal 100A and the display device 200 . Therefore, after receiving the zoom instruction, the display device 200 can move the picture according to the display position, so that the zoom position coincides with the center position of the display 275, so that in the subsequent zoom process, the zoomed image area can be displayed in the center area, achieve better display effect. That is, after moving the position of the target picture, the display device 200 uses the display position as a reference, and performs scaling on the target picture according to the scaling ratio.

It can be known from the above technical solutions that the picture synchronous scaling method provided in the above embodiments can receive the scaling instruction input by the user through the mobile terminal 100A, and perform scaling on the original picture according to the scaling instruction. When the boundary of the original picture exceeds the boundary of the display area, a zoom instruction is sent to the display device 200 . The scaling instruction includes a display position and a scaling ratio, and the display device 200 can perform synchronous scaling on the target picture according to the display position and the scaling ratio. The method realizes the synchronous scaling of pictures through the mapping algorithm based on the display position and the scaling ratio, and alleviates the problem of display misalignment.

Based on the above-mentioned picture synchronous scaling method, in some embodiments of the present application, a mobile terminal 100A is further provided, which includes a display unit 310 , a communication unit 320 and a processing unit 330 . Wherein, the display unit 310 is configured to display the original picture; the communication unit 320 is configured to establish a communication connection with the display device; the processing unit 330 is configured to execute the following program steps:

Receive a zoom command input by the user; in response to the zoom command, perform zooming on the displayed original picture; if the boundary of the original picture exceeds the display area boundary of the display unit 310, calculate the display position and the zoom ratio; The display device sends a zoom instruction, where the zoom instruction includes the display position and the zoom ratio.

Similarly, in order to cooperate with the mobile terminal 100A to implement picture synchronization scaling, in some embodiments, a display device 200 is also provided, including a display 275 , a communicator 220 and a controller 250 . Wherein, the display 275 is configured to display the target picture, the communicator 220 is configured to establish a communication connection with the mobile terminal 100A; the controller 250 is configured to execute the following program steps:

receiving a zoom instruction, the zoom instruction is input by the mobile terminal 100A when the border of the original image exceeds the border of the display area, and the zoom instruction includes a display position and a zoom ratio; the original image is the image displayed in the mobile terminal 100A; In response to the zoom instruction, move the position of the target picture according to the display position, where the target picture is a picture displayed in the display; and use the display position as a reference, perform the execution on the target picture according to the zoom ratio zoom.

According to the mobile terminal 100A and the display device 200 described above, in some embodiments of the present application, the processing unit 330 and the controller 250 respectively execute the configured program steps to implement the picture synchronization scaling method in the above embodiments. After the communication connection is established, the user can manipulate the mobile terminal 100A to zoom the original picture displayed on the mobile terminal 100A, and after the original picture exceeds the zoom area, the communication unit 320 sends a zoom instruction to the display device 200 . After receiving the scaling instruction through the communicator 220, the controller 250 of the display device 200 can perform moving and scaling processing on the target picture, so as to realize the synchronous processing with the scaling process in the mobile terminal 100A.

< Scaling processing combined with display area >

Since the mobile terminal 100A displays the original image and the display device 200 displays the target image, the image will be zoomed and displayed according to different screen sizes, but in practical applications, as long as the mobile terminal 100A zooms the original image to the same size as the display device After the 200 has been zoomed to the same degree, performing the zooming will make the display device 200 need to display the picture locally. Therefore, in some embodiments, the zooming process can also be judged comprehensively according to the screen size of the mobile terminal 100A and the screen size of the display device. , to determine the zoom boundary. That is, after establishing the picture projection relationship, the mobile terminal 100A may first detect its own screen state and the screen state of the display device 200, and determine the zoom boundary of the display area according to the detection result. Therefore, the processing unit 330 of the mobile terminal 100A may also be configured to perform the following program steps:

Get the display area aspect ratio. The aspect ratio of the display area includes the aspect ratio of the display area of the display unit 310 and the aspect ratio of the display area of the display device. In practical applications, in order to obtain the aspect ratio of the display area of the display unit 310, the processing unit 330 may detect the screen state of the mobile terminal 100A in real time to determine the aspect ratio of the display area. Due to the limitation of the interface UI, not all display areas on the screen are used to display pictures. Therefore, in some embodiments of the present application, the display area refers to the area used to display pictures in the interface. The display area aspect ratio refers to the aspect ratio of the area used to display the picture, see formula (1):

wh_mobile=W_mobile/H_mobile Formula (1)

In the formula, wh_mobile is the aspect ratio of the display area of the mobile terminal 100A; W_mobile is the width of the picture display area in the mobile terminal 100A; H_mobile is the height of the picture display area in the mobile terminal 100A.

Obviously, for different placement states, the mobile terminal 100A usually has picture display areas with different shapes. The processing unit 330 may determine the shape of the display area by detecting the placement state of the mobile terminal 100A, and then calculate the aspect ratio of the display area. For example, it can be determined whether the mobile terminal 100A is currently in a landscape state or a portrait state by detecting a gravity sensor built in the mobile terminal 100A. When the mobile terminal 100A is in the landscape state, the width of the display area is greater than the height, that is, the aspect ratio of the display area is greater than 1; when the mobile terminal 100A is in the portrait state, the height of the display area is greater than the width, that is, the aspect ratio of the display area is less than 1 . It should be noted that, in the process of calculating the aspect ratio of the display area, the processing unit 330 may calculate the actual physical size of the width and height of the display area, or calculate the number of pixels occupied by the width and height of the display area. calculation, the actual calculation result is the same.

The aspect ratio of the display area of the display device 200 needs to be acquired from the display device 200 after the mobile terminal 100A establishes a communication connection relationship with the display device 200 . For example, when the aspect ratio of the display area needs to be determined, the processing unit 330 may send an acquisition instruction to the display device 200 through the communication unit 320 . After receiving the acquisition instruction, the controller 250 of the display device 200 may respond to the acquisition instruction and extract the width and height of the display area currently on the display 275, thereby feeding back the width and height data of the display area to the mobile terminal 100A. The processing unit 330 of the mobile terminal 100A then calculates the aspect ratio of the display area of the display device 200 according to the feedback result. The calculation method of the display area aspect ratio wh_tv of the display device 200 is the same as the calculation method of the mobile terminal 100A, and will not be repeated.

Similarly, for different types of display devices 200, they have different shapes of display areas. In addition, for the display device 200 with a part of the rotatable display 275, there are display areas with different shapes in different rotation states, so it is necessary to determine the aspect ratio of the display area according to different rotation states in practical applications. For example, when the display device 200 is in a landscape state, the display area aspect ratio wh_tv is greater than 1; when the display device 200 is in a portrait state, the display area aspect ratio wh_tv is less than 1.

After obtaining the display area aspect ratio, the process may compare the display area aspect ratio of the display unit 310 with the display area aspect ratio of the display device to determine the zoom boundary. If the aspect ratio of the display area of the display unit 310 is greater than the aspect ratio of the display area of the display device 200 , the left and right sides of the display unit 310 are determined as the scaling boundaries. That is, if wh_tv<wh_mobile, the zoom boundary of the mobile terminal 100A is the left and right boundaries of the display area in the display unit 310 , that is, when the picture on the mobile terminal 100A fills the display area on the left and right, it is considered to be the size of the original image, and if it continues to zoom in, it starts to display The device 200 sends a zoom-in command; when the left and right boundaries are not reached, no zoom-in command is sent (or a zoom command is sent, but the information in the zoom command instructs the display device 200 to maintain the original image state).

If the aspect ratio of the display area of the display unit 310 is smaller than the aspect ratio of the display area of the display device, determine that the upper and lower sides of the display unit 310 are the scaling boundaries. That is, if wh_tv>wh_mobile, the zoom boundary of the mobile terminal 100A is the upper and lower boundaries of the display area, that is, when the picture on the mobile terminal 100A fills the display area up and down, it is considered to be the size of the original image, and if it continues to zoom in, it starts to send the zoom to the display device 200 command; when the start boundary is not reached, no zoom command is sent.

It can be seen that, in some embodiments of the present application, by comparing the aspect ratio of the display area of the display unit 310 with the aspect ratio of the display area of the display device 200 , a scaling boundary can be determined, and a scaling instruction can be sent to the display device 200 according to the scaling boundary control. The aspect ratio of the display area can be adapted to various types of screens of the mobile terminal 100A and the display device 200 , thereby alleviating the problem of excessively different display effects caused by differences in screen sizes.

After determining the zoom boundary, the processing unit 330 of the mobile terminal 100A will automatically send a zoom instruction to the display device 200 when it is determined that the displayed picture exceeds the zoom boundary. Since the sent zoom command includes the display position and the zoom ratio, but because of the difference in screen size between the mobile terminal 100A and the display device 200 , the corresponding zoom position cannot be directly sent to the display device 200 when the zoom command is input in the mobile terminal 100A , and needs to undergo certain conversion to convert the zoom position into a specific area on the picture, and then send it to the display device 200, so that the display device 200 can zoom in and out according to the display position, so as to alleviate the problem of display misalignment.

To this end, the processing unit 330 of the mobile terminal 100A also needs to calculate the display position and the zoom ratio before sending the zoom command. In some embodiments, the display position can be calculated according to the following formula, see formula (2) (3):

Ox=(W/2-(X0-X1))/W×Px Formula (2);

Oy=(H/2-(Y0-Y1))/W×Py Formula (3);

In the formula, (Ox, Oy) is the coordinate of the described display position; W is the width of the picture after scaling; H is the height of the picture after scaling; (X0, Y0) The coordinates of the center point of the display area; (X1, Y1) are the scaling The coordinates of the center point of the rear image; Px is the pixel value in the width direction of the image; Py is the pixel value in the height direction of the image.

In some embodiments, the scaling ratio can be calculated according to the following formula, see formula (4) (5):

Scale=H1/H0 Formula (4);

Or, Scale=W1/W0 Formula (5);

In the formula, Scale is the scaling ratio; H0 is the height of the display area; H1 is the height of the image after scaling; W0 is the width of the display area; W1 is the width of the image after scaling.

In some embodiments, FIG. 10 is a schematic diagram of display position calculation according to one or more embodiments of the present application. As shown in FIG. 10 , the center point (X0, Y0) of the display area of the mobile phone in the figure, if the display resolution of the original image The ratio is 1920×1080, and the zoom position is 40% from left to right and 40% from top to bottom of the enlarged image, corresponding to pixels (1920×40%), (1080×40%), namely (768, 432). Then the information sent to the display device is, the picture display position is (768, 432), and the magnification is 2 times. Set the display position to be sent to the display device as (Ox, Oy) and the scaling ratio as Scale, taking the case of wh_tv>wh_mobile as an example, the scaling boundary of the mobile terminal 100A is the upper and lower boundaries. In any reference coordinate system of the screen (x in the horizontal direction, y in the vertical direction), for the mobile terminal 100A, the coordinates of the A/B/C/D/E/F points can be directly obtained, and the picture width and height pixels Px, Py are also known, then the scaling ratio, see formula (6):

Scale=(Cy-Ay)/(Fy-Dy) Formula (6);

Among them, Cy, Ay, Fy, Dy represent the y-axis coordinate values of C, A, F, and D respectively, that is, the height of the display area H0=Cy-Ay; the height of the image after scaling H1=Fy-Dy. And, calculate the coordinates of the center point G(X0, Y0) of the display area and the coordinates of the center point H(X1, Y1) of the zoomed image respectively, see formulas (7) (8):

X0=Gx=(Dx+Ex)/2; Y0=Gy=(Dy+Ey)/2; Formula (7)

X1=Hx=(Ax+Bx)/2; Y1=Hy=(Ay+By)/2 Formula (8)

Therefore, the enlarged picture size: H=AC=Cy-Ay; W=AB=Bx-Ax.

Then, Hx-Gx and Hy-Gy in the following formula are the moving distances of the center point of the enlarged image in the x-axis and y-axis directions compared with the center point of the original image, see formula (9) (10):

Ox=(AB/2-(Hx-Gx))/AB×Px Formula (9)

Oy=(AC/2-(Hy-Gy))/AC×Py Formula (10)

To sum up, after the mobile terminal 100A calculates and obtains the display position and the scaling ratio, it can transmit the calculated (Ox, Oy) and Scale to the display device 200 through the data transmission channel, so that the display device 200 can perform the calculation according to the transmitted display position and scale. The zoom ratio moves and zooms the displayed picture.

< Picture moves first, then zooms >

In some embodiments, in order to adapt to the screen form of the display 275 of the display device 200 , the display device 200 needs to move the displayed picture so that the picture is in an appropriate position, and then perform zooming, so that the area zoomed by the user is on the display 275 Therefore, the step of moving the position of the target image according to the display position further includes the following steps:

moving the target picture to a default position; traversing the pixel points corresponding to the display position in the target picture; moving the target picture to set the pixel points to coincide with the center point of the display. After receiving the zoom instruction, the display device 200 can respond to the zoom instruction through the controller 250, extract the display position and zoom ratio in the zoom instruction, and execute the relevant display control program, first reset the picture to the default display position, Then move the (Ox, Oy) pixel position of the picture to the center of the screen, and then scale the picture by Scale times from the center of the screen. The default position may be an initial display position determined according to the last moving or zooming process, or may be a designated position on the UI interface in the display device 200 . By moving the picture to the default position, the area corresponding to the zoomed position in the picture can be easily identified, thereby facilitating synchronous zooming.

With the zooming process performed on the display device 200, the picture displayed on the display 275 will be resized according to the zooming instruction. In some cases, the enlarged picture may have some edges beyond the display area. In order to obtain a better display Effect. In some embodiments, FIG. 11 is a schematic diagram illustrating the effect of adjusting the position of a picture toward the border by a display device according to one or more embodiments of the present application, and FIG. 12 is a schematic diagram illustrating the effect of adjusting the position of a picture in the center of the display device according to one or more embodiments of the present application. 11 and 12, the step of performing scaling on the target picture according to the scaling ratio based on the display position further includes: detecting the target picture boundary after performing the scaling; judging the target picture boundary Whether it exceeds the display area boundary of the display; if the target picture boundary exceeds the display area boundary of the display, move the zoomed target picture position to make the target picture boundary coincide with the display area boundary; if the target The picture boundary does not exceed the display area boundary of the display, and the position of the target picture is moved so that the target picture is displayed in the center. FIG. 13 is a schematic diagram of a synchronous zoom effect when a picture does not exceed the border of the display area according to one or more embodiments of the present application; FIG. 14 is a schematic diagram of a synchronized zoom effect when a picture exceeds the border of the display area according to one or more embodiments of the present application 13 and 14 , after the display device 200 performs zooming on the picture, it can detect the boundary of the target picture to determine whether the boundary of the target picture exceeds the boundary of the display area. If the display area of the display device 200 can exceed the display area on one or two boundaries at this time, the boundary is adjusted to be exactly at the position of the display boundary to display larger picture content. If the enlarged picture does not reach the border on both sides, it will automatically move the picture to the middle position for display.

It can be known from the above embodiments that the mobile terminal 100A and the display device 200 can be synchronously mapped and scaled in the case of wh_tv>wh_mobile. During the entire process of synchronous zooming, the display position is used to determine the association between the pictures displayed in the mobile terminal 100A and the display device 200, and the zoom position specified by the user on the mobile terminal 100A is mapped to the display device 200, so as to scale according to the corresponding zoom position. After the scaling is performed, the scaling of the picture displayed in the display device 200 is completed. It not only realizes synchronous scaling, but also can adapt to the difference in screen form between the mobile terminal 100A and the display device 200, so as to alleviate the problem of display misalignment.

It should be noted that the mobile terminal 100A may perform sampling during the entire zooming process of the user's operation, and perform multiple calculations and send zooming instructions during the entire zooming process, so as to achieve a smooth operation effect. The method adopted in the above embodiment is also applicable to the display process of the vertical picture when wh_tv>wh_mobile. However, if wh_tv<wh_mobile, that is, when the display device 200 is in the vertical screen mode or the mobile terminal 100A is in the horizontal state, the synchronous scaling method should be based on the left and right boundaries of the display area when the mobile terminal 100A selects the enlargement boundary, and the rest of the calculation principles are the same and will not be repeated here.

The synchronous display between the mobile terminal 100A and the display device 200 may also be completed by means of screen projection. That is, in some embodiments, the mobile terminal 100A may directly send the displayed content to the display device 200 through the established communication connection, and the display device 200 directly displays the screencast content. At this time, when the user inputs the zoom action instruction, since the content displayed on the mobile terminal 100A changes with the zoom action instruction, the screen projection image sent to the display device 200 also changes, realizing the effect of synchronous zooming.

In order to realize the synchronous scaling of pictures between the mobile terminal 100A and the display device 200, a method for synchronous scaling of pictures is also provided in some embodiments of the present application, including the following steps:

A zoom instruction sent by the mobile terminal 100A is received. Similar to the above embodiment, before executing the picture synchronous scaling method, the mobile terminal 100A first establishes a communication connection relationship with the display device 200, and sends a scaling instruction to the display device 200 after the communication connection is established. The display device 200 may receive the scaling instruction through the communicator.

In response to the zoom instruction, a target picture is displayed in a preset display area of the display. FIG. 15 is a schematic diagram of a preset display area according to one or more embodiments of the present application, and FIG. 16 is a schematic diagram of an effect of zooming a target image in the preset display area according to one or more embodiments of the present application, as shown in FIGS. 15 and 16 . As shown, after receiving the zoom instruction, the display device 200 can construct a preset display area on the display 275 according to the zoom instruction, and the preset display area covers the entire display area in the mobile terminal 100A. For example, after the user executes the zoom-in command on the mobile terminal 100A, the display device 200 may construct a preset display area sufficient to cover the mobile terminal 100A, so that the picture is displayed in the preset display area.

The preset display area can be consistent with the display area in the mobile terminal 100A in the form of boundaries, and when the displayed contents are the same, the positions of the displayed contents in each area in the display area are also consistent, so that the execution of the mobile terminal 100A can be performed. The zoom command is directly passed to the display device 200 . That is, the display device 200 may perform synchronous scaling on the target picture according to the scaling instruction.

Because in some embodiments of the present application, by constructing a preset display area covering all display areas in the mobile terminal 100A in the display device 200, the display position relationship of the picture in the mobile terminal 100A can be preserved, so that the zooming can be directly adjusted. The position and scaling ratio are applied in the scaling instruction, so that the display device 200 can directly perform the scaling action according to the scaling position and scaling ratio, thereby avoiding the problem of display misalignment.

Based on the above picture synchronization scaling method, some embodiments of the present application further provide a display device 200 including a display 275 , a communicator 220 and a controller 250 . Wherein, the display 275 is configured to display the target picture, the communicator 220 is configured to establish a communication connection with the mobile terminal 100A; the controller 250 is configured to execute the following program steps:

Receive a zoom instruction sent by the mobile terminal 100A; in response to the zoom instruction, display a target image in a preset display area of the display, and the preset display area covers the entire display area in the mobile terminal 100A ; According to the scaling instruction, perform synchronous scaling on the target picture. Since the overall screen sizes of the mobile terminal 100A and the display device are quite different, the zoom command input by the user on the mobile terminal 100A may cause data deviation when directly applied to the display device 200 . For example, a user enters a two-finger touch gesture separated from each other on a mobile phone to enter a zoom command. Due to the limitation of the finger length and the screen of the mobile phone, the separation distance is generally 1-10cm. Since the actual size of the picture displayed by the display device 200 is larger, the separation distance of 1-10cm corresponds to the mobile terminal 100A and the display device 200 . The scaling ratios are different, so it is easy to break the synchronization of the picture scaling process.

To this end, in some embodiments of the present application, before sending the zoom instruction to the display device 200, the mobile terminal 100A may further perform the following steps:

Receives zoom commands entered by the user. In this embodiment, the mobile terminal 100A can receive the zoom command input by the user in the manner in the above-mentioned embodiment. Similarly, the zoom command can include the zoom position and the zoom ratio, and supports different ways to complete the input of the zoom command. In response to the zoom command, an adjustment displacement and an adjustment center point are calculated. After receiving the zoom command input by the user, the mobile terminal 100A may calculate the adjustment displacement and adjust the center point according to the zoom command. The adjustment distance is used to indicate the zoom ratio, and may refer to the distance in the control action determined according to the zoom command input by the user, such as the separation distance of two fingers. The adjustment center of gravity point is the point used to mark the zoom position, for example, the midpoint of the line connecting the two-finger touch points.

It should be noted that, in practical applications, in order to realize the synchronous control of the picture viewing process, the user may input a movement command in addition to the zoom command when displaying the picture, such as displaying partial content in the picture by sliding touch. . For such a movement command, the mobile terminal 100A may send the movement position point and the movement distance to the display device 200, so that the display device 200 can move the picture synchronously.

Perform scaling of the displayed original image by adjusting the displacement and adjusting the center point as described. After calculating the adjustment displacement and the adjustment center point, the mobile terminal 100A can determine the zoom position according to the adjustment center point on the one hand, and determine the zoom ratio according to the adjustment displacement on the other hand, and perform zooming on the displayed original picture. At the same time, the mobile terminal 100A also sends a zoom instruction to the display device 200 according to the adjustment displacement or the center point adjustment, where the zoom instruction includes adjusting the displacement or adjusting the center point.

Based on the above method, in some embodiments, a mobile terminal 100A is also provided, which includes a display unit 310 , a communication unit 320 and a processing unit 330 . Wherein, the display unit 310 is configured to display the original picture; the communication unit 320 is configured to establish a communication connection with the display device; the processing unit 330 is configured to execute the following program steps:

Receive a zoom command input by the user; in response to the zoom command, calculate the adjustment displacement and adjust the center point; perform zooming on the displayed original picture according to the adjustment displacement and adjustment center point; send a zoom instruction to the display device, the Scaling commands include adjusting the displacement or adjusting the center point. Due to the difference in screen size between the mobile terminal 100A and the display device 200, after the zoom instruction is sent to the display device 200, the display device 200 can also adjust the zoom parameter according to the adjustment displacement or the center point in the zoom instruction, so as to perform the operation on the picture. zoom.

In some embodiments, according to the scaling instruction, the step of performing synchronous scaling on the target picture further includes: parsing the scaling instruction to obtain an adjustment displacement; according to the display area in the mobile terminal 100A and the preset The proportional relationship of the display area is to convert the adjusted displacement into a scaled size; and the picture in the preset display area is scaled according to the scaled size.

In this embodiment, after receiving the scaling instruction, the controller 250 may first parse the scaling instruction to obtain the adjustment displacement, and then convert the adjustment displacement into scaling according to the proportional relationship between the mobile terminal 100A and the display area in the display device 200 size, so as to perform scaling on the picture in the preset display area according to the scaling size.

For example, when the screen size (in pixels) of the display 275 of the display device 200 is 5 times the screen size (in pixels) of the display unit 310 of the mobile terminal 100A, the zoom size parsed in the zoom instruction is L0, then the zoom The scale is converted into 5×L0, and the scaled size can be obtained.

In some embodiments, the step of performing synchronous scaling on the target picture further includes: parsing the scaling instruction to obtain an adjustment center point; locating the center point of the target picture; moving the target picture to make the target picture The center point coincides with the adjustment center point; and based on the center point of the target image, zooming is performed on the image in the preset display area. In order to determine the zooming position of the picture on the display device 200, after the controller 250 obtains the zooming instruction, it can parse the zooming instruction to obtain the adjustment center point, locate the center point of the displayed target image, and move the target image to Make the center point of the target image coincide with the adjustment center point, even if the zoomed position is in the middle area of the display 275, and then use the center point of the target image as a reference to perform zooming on the image in the preset display area.

It can be seen that, in the above embodiment, the display device 200 can determine the scaling ratio and the scaling displacement of the image displayed by the display device 200 according to the adjustment displacement and the adjustment center point respectively, so as to convert the zoom command received by the mobile terminal 100A into an image suitable for display A scaling instruction of the device 200 to accommodate the difference in screen size between the mobile terminal 100A and the display device 200 .

As can be seen from the above technical solutions, the picture synchronous scaling method provided by the above embodiments can, after receiving the scaling command, the mobile terminal 100A can calculate the adjustment displacement and adjust the center point according to the scaling command, so as to scale the original picture. At the same time, the mobile terminal 100A also sends a zoom instruction to the display device, so that the display device synchronously zooms the target picture in the preset display area covering the entire display area of the mobile terminal 100A. By constructing a preset display area covering the entire display area of the mobile terminal 100A in the display device, the method realizes the synchronous conversion of the zoom command executed on the mobile terminal 100A to the display device, realizes the synchronous zooming of the picture, and alleviates the problem of display misalignment.

In order to realize the synchronous scaling of pictures between the mobile terminal 100A and the display device 200, some embodiments of the present application also provide a synchronous picture scaling method, including the following steps: the mobile terminal 100A receives a scaling command input by a user; The terminal 100A calculates the reference point and the adjustment parameter according to the zoom command, and the adjustment parameter is the moving distance and/or the zoom ratio calculated and obtained by the mobile terminal 100A according to the user input; the mobile terminal 100A calculates the reference point and the adjustment parameter according to the parameters, perform scaling on the displayed original picture and send a scaling instruction to the display device 200, where the scaling instruction includes the reference point and the adjustment parameter; the display device 200, in response to the scaling instruction, obtains the current display of the target picture state; the display device 200 scales the target picture on the basis of the current display state according to the reference point and the adjustment parameter.

The difference from the above embodiments is that in some embodiments of the present application, the mobile terminal 100A directly sends the reference point and adjustment parameters to the display device 200 , and the display device 200 can scale the currently displayed picture according to the reference point and the adjustment parameters. That is, the operation on the mobile terminal 100A is copied to the display device 200, and the mobile terminal 100A can send a zoom command to the display device 200, and the zoom command represents the adjustment parameter amount based on the current display of the mobile terminal 100A, that is, the movement based on the current display state The parameter amount of the image, and/or the amount of image scaling based on the fiducial point.

Based on this, if the initial picture display effects of the mobile terminal 100A and the display device 200 are corresponding, the result of the previous state operation at each step is also corresponding. According to the recursive induction, the results of each subsequent operation are both corresponding. That is, in some embodiments of the present application, sufficient information for the mobile terminal 100A and the display device 200 to correspond to synchronous scaling is: if the operation performed on the picture in the mobile terminal 100A is to move, the display device 200 is notified in which direction it has moved. How much distance; if the operation performed on the picture in the mobile terminal 100A is zooming, the display device 200 is notified based on which point and how many times it is zoomed.

Based on the above picture synchronization scaling method, a display device 200 in some embodiments of the present application includes a display 275 , a communicator 220 and a controller 250 . Wherein, the display 275 is configured to display the target picture, the communicator 220 is configured to establish a communication connection with the mobile terminal 100A; the controller 250 is configured to execute the following program steps:

Receive a zoom instruction sent by the mobile terminal 100A; the zoom instruction includes a reference point and an adjustment parameter, and the adjustment parameter is the moving distance and/or zoom ratio calculated and obtained by the mobile terminal 100A according to the user input; in response to the The zoom instruction is used to obtain the current display state of the target picture; and according to the reference point and the adjustment parameter, the target picture is zoomed on the basis of the current display state.

Based on the above method, in some embodiments, a mobile terminal 100A is also provided, which includes a display unit 310 , a communication unit 320 and a processing unit 330 . Wherein, the display unit 310 is configured to display the original picture; the communication unit 320 is configured to establish a communication connection with the display device 200; the processing unit 330 is configured to execute the following program steps:

Receive a zoom command input by a user; in response to the zoom command, calculate a reference point and an adjustment parameter, where the adjustment parameter is the moving distance and/or zoom ratio calculated and obtained by the mobile terminal 100A according to the user input; according to the reference point and an adjustment parameter, perform zooming on the displayed original picture; send a zooming instruction to the display device, the zooming instruction reference point and the adjustment parameter.

As described above, to implement the screen projection function, a communication connection needs to be established between the mobile terminal 100A and the display device 200 , for example, through a WiFi connection. Taking P2P (Peer To Peer) screen projection as an example, when a WiFi connection is used, the mobile terminal 100A and the display device 200 are connected to the same wireless network, and the P2P service is used to realize the screen projection. At the same time, the normal WLAN must be ensured. At this time, the problem of P2P preempting WLAN resources will occur, which will limit the normal WLAN function and increase the packet loss rate.

In some embodiments, FIG. 17 is a possible P2P device discovery process according to one or more embodiments of the present application. As shown in FIG. 17 , in the device discovery stage of P2P pairing, the P2P mobile terminal 100A and The display device 200 should constantly switch between scanning and monitoring on channels 1, 6, and 11. When the mobile terminal 100A and the display device 200 happen to be in the same channel, and the mobile terminal 100A scans, that is, sends a detection request, and displays the When the device 200 monitors the detection request sent by the mobile terminal 100A, the display device 200 responds to the detection request and replies to the detection response to complete the device discovery process (this process can also be performed by the display device 200 under the same channel). scanning, the mobile terminal 100A monitors). Based on the above process requirements of the device discovery stage, if the user is to perform screen projection at any time, the background of the display device 200 needs to continuously perform P2P scanning and monitoring, which will cause the channel to keep on 1, 6, 11. Switching greatly affects the normal use of WLAN ports. It not only occupies antenna resources, but also frequently switches channels, which increases network latency and makes the network more stuck. To this end, the following solutions are proposed, multiple working modes are set, and corresponding working modes are selected according to different situations.

In some embodiments, FIG. 18 is a process of adjusting a duration in a network transmission-based mode according to one or more embodiments of the present application. As shown in FIG. 18 , when some network-dependent applications such as download applications , audio applications, video applications, etc. request network data, the network data request is transmitted to the driver module through the framework layer and the core layer, and the driver module determines that the transmission mode of the display device is the network-based mode, and sends The first transmission duration adjustment command is sent to wireless communication firmware such as WiFi firmware.

In some embodiments, the first transmission duration adjustment instruction may be a specific duration allocation instruction, or may be a Δt adjustment instruction as shown in FIG. 8 , where Δt is the difference between the time slice occupied by network transmission and screencast transmission. difference. Taking the transmission time slice provided by the wireless communication module as 1s as an example, when the first transmission duration adjustment command is Δt=800, the WiFi firmware determines the first transmission duration to provide transmission services for network transmission according to the first transmission duration adjustment command: 900ms, and the second transmission duration for providing transmission services for screencast transmission is 100ms. Therefore, network transmission is performed within 900ms of each second, and the WLAN port transmits and receives data packets, and the data packets are transmitted to the network application through the driver module, core layer and framework layer.

In some embodiments, the display device further includes a detection module 225 for detecting network transmission quality and triggering the driving module according to the transmission quality. In the network transmission-based mode, the detection module is the drive module. When the network environment becomes worse, the detection module detects that the WiFi packet loss rate becomes higher and the delay is higher than the preset threshold, and sends the packet loss command to the drive module, and the drive module can adjust the command to the size of Δt through the third transmission duration Make mandatory adjustment and gradually adjust Δt, for example, gradually increase the absolute value of Δt in units of 10ms, and continuously compare the delay of WLAN until it is adjusted to the level that satisfies the basic sending and receiving packets of WLAN (for example, the delay is lower than 200ms) , and at the same time ensure that the time that P2P occupies the antenna will not affect the normal monitoring of P2P, for example, not less than 50ms/1s. For example, when the driver module detects that the delay is higher than the preset threshold and adjusts Δt=810, the WiFi firmware determines that the first transmission duration for network transmission is 905ms, and the second transmission duration for screencast transmission is determined to be 905ms. The transmission time is 95ms. If the delay is still higher than the preset threshold, adjust Δt=820, and so on.

In some embodiments, FIG. 20 is a possible P2P device discovery process according to one or more embodiments of the present application. As shown in FIG. 20 , combined with the usage scenario of the display device, the P2P discovery device of the display device is The scanning and monitoring of the stage is changed to monitoring only on fixed channels (such as 1, 6, and 11), which can reduce the occupation of antenna resources caused by channel switching. At the same time, because the display device is always in the monitoring state, it can also improve P2P discovery. equipment efficiency.

In some embodiments, FIG. 19 is a process of adjusting a duration in a mode mainly based on screen projection transmission according to one or more embodiments of the present application. As shown in FIG. 19 , when a transmission service is provided for screen projection transmission Within the second transmission duration, such as within 100ms of the example in Figure 20, when the screen-casting application initiates P2P monitoring, the monitoring command is transmitted to the WiFi firmware via the framework layer, the adaptation layer and the driver module, and the WiFi firmware switches to the monitoring channel such as channel 6 to monitor.

In some embodiments, when the WiFi firmware is monitoring, a P2P detection request is received, and the detection request is sent to the driver module, and the driver module parses the message containing the P2P IE information, thereby determining that the transmission mode of the display device is Screen-based mode.

In some embodiments, the driving module sends a second transmission duration adjustment instruction to the WiFi firmware after determining that the transmission mode in which the display device is located is the screen-casting mode. In some embodiments, the second transmission duration adjustment instruction may be a specific duration allocation instruction, or may be a Δt adjustment instruction as shown in FIG. 19 , where Δt is the difference between the time slice occupied by network transmission and screencast transmission. difference. Taking the transmission time slice provided by the wireless communication module as 1s as an example, when the second transmission duration adjustment command is Δt=-600, the WiFi firmware determines the first transmission duration to provide transmission services for network transmission according to the duration adjustment command to be 200ms, which is The second transmission duration for the screencast transmission to provide the transmission service is 800ms.

Therefore, the screen projection transmission is performed within 800ms of each second, and the P2P port sends and receives data packets, and the data packets are transmitted to the screen projection application through the driver module, the adaptation layer and the framework layer. If it is video data, it can be played through the screen casting application.

In some embodiments, in the mode of screencasting transmission as the main mode, the detection module is a screencasting application. When the screencasting application detects that the P2P packet loss rate has increased and exceeds the preset threshold, for example, the number of lost packets exceeds 100 packets, the screencasting will show a vase freeze problem. The screen-casting application will send the packet loss command to the driver module. The driver module can forcefully adjust the size of Δt through the third transmission duration adjustment command, and gradually adjust Δt. Constantly compare the packet loss rate of the projection screen until it is adjusted to meet the level of basic P2P sending and receiving packets, and at the same time ensure that the time that the WLAN occupies the antenna will not affect the normal progress of network transmission, for example, not less than 50ms/1s.

For example, when the driver module receives the packet loss command and adjusts Δt=-610, the WiFi firmware determines that the first transmission duration for network transmission is 195ms, and the second transmission duration for screencast transmission is 805ms. If the delay is still higher than the preset threshold, adjust Δt=-620, and so on.

The above embodiment takes P2P screen projection as an example, and can also be applied to other screen projection technologies, which is not limited thereto.

By dividing the network transmission and screen projection transmission into time slices, the network transmission and the screen projection transmission work in their respective time slices, thus solving the problem of competition between the two for resources and ensuring the orderly and orderly network transmission and screen projection. At the same time, the working efficiency of the display device is improved. At the same time, fully considering the actual working conditions of the display device, two transmission modes are set: the network transmission-based mode and the point-to-point projection screen transmission mode, and the transmission duration adjustment instructions are determined according to different transmission modes, so that network transmission and projection The working time slice of the screen can be flexibly adjusted with different transmission modes, which further realizes the effective use of resources.

For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion in some embodiments is not intended to be exhaustive or to limit implementations to the specific forms disclosed above. Numerous modifications and variations are possible in light of the above teachings. The above embodiments have been chosen and described to better explain the principles and practical applications, so as to enable those skilled in the art to better utilize the embodiments and various modified embodiments suitable for specific use considerations.

Claims

A display device comprising:

monitor;

a communicator configured to establish a communication connection with the mobile terminal;

Controller, configured as:

establishing a screen projection relationship with the mobile terminal;

Displaying a target picture through the display, where the target picture is a picture with the same content as the original picture displayed on the display;

When the boundary of the original picture exceeds the boundary of the display area of the mobile terminal, the target picture is scaled synchronously.
The display device of claim 1, the controller is further configured to:

In the step of synchronously scaling the target picture, a scaling instruction is received, and the scaling instruction is input by the mobile terminal when the border of the original picture exceeds the border of the display area, and the scaling instruction includes a display position and a scaling ratio;

In response to the zoom instruction, move the position of the target picture according to the display position;

Based on the display position, the target picture is zoomed according to the zoom ratio.
The display device of claim 2, the controller is further configured to:

In the step of moving the position of the target picture according to the display position, the target picture is moved to a default position;

Traverse the pixel points corresponding to the display position in the target picture;

The target picture is moved to set the pixel points to coincide with the center point of the display.
The display device of claim 2, the controller is further configured to:

Taking the display position as a benchmark, in the step of performing scaling on the target picture according to the scaling ratio, detecting the target picture boundary after performing the scaling;

Determine whether the target picture boundary exceeds the display area boundary of the display;

If the boundary of the target picture exceeds the boundary of the display area of the display, the position of the scaled target picture is moved to make the boundary of the target picture coincide with the boundary of the display area.
The display device of claim 4, the controller is further configured to:

Taking the display position as a benchmark, in the step of scaling the target picture according to the scaling ratio, if the boundary of the target picture does not exceed the display area boundary of the display, move the target picture position so that the target picture is in the desired position. The target image is displayed in the center.
The display device of claim 1, the controller is further configured to:

In the step of synchronously scaling the target picture, receiving a scaling instruction sent by the mobile terminal;

In response to the zoom instruction, displaying the target picture in a preset display area of the display, the preset display area covering all the display areas in the mobile terminal;

According to the scaling instruction, synchronous scaling is performed on the target picture.
The display device of claim 6, the controller is further configured to:

In the step of performing synchronous scaling on the target image according to the scaling instruction, parsing the scaling instruction to obtain an adjustment displacement;

According to the proportional relationship between the display area in the mobile terminal and the preset display area, the adjusted displacement is converted into a scaled size in equal proportions;

Zooming is performed on the picture in the preset display area according to the zooming size.
The display device of claim 6, the controller is further configured to:

In the step of performing synchronous scaling on the target image according to the scaling instruction, parsing the scaling instruction to obtain an adjustment center point;

Locate the center point of the target image;

moving the target picture so that the center point of the target picture coincides with the adjustment center point;

Based on the center point of the target picture, zooming is performed on the pictures in the preset display area.
The display device of claim 1, the controller is further configured to:

In the step of synchronously scaling the target picture, a scaling instruction sent by the mobile terminal is received, and the scaling instruction includes a reference point and an adjustment parameter, and the adjustment parameter is the moving distance calculated by the mobile terminal according to the user input and / or scaling;

In response to the zoom instruction, obtain the current display state of the target picture;

According to the reference point and the adjustment parameter, scaling is performed on the target picture based on the current display state.
A mobile terminal, comprising:

Display unit;

a communication unit, configured to establish a communication connection with the display device;

Processing unit, configured as:

Receive the zoom command input by the user;

In response to the zoom command, zooming is performed on the displayed original picture;

If the border of the original picture exceeds the border of the display area of the display unit, calculate the display position and the zoom ratio;

Send a zoom instruction to the display device, where the zoom instruction includes the display position and the zoom ratio.
The mobile terminal of claim 10, the processing unit is further configured to:

In the step of calculating the display position and the zoom ratio, the aspect ratio of the display area is obtained, and the aspect ratio of the display area includes the aspect ratio of the display area of the display unit and the aspect ratio of the display area of the display device;

comparing the aspect ratio of the display area of the display unit and the aspect ratio of the display area of the display device to determine the scaling boundary;

If the aspect ratio of the display area of the display unit is greater than the aspect ratio of the display area of the display device, determine that the left and right sides of the display unit are the scaling boundaries;

If the aspect ratio of the display area of the display unit is smaller than the aspect ratio of the display area of the display device, determine that the upper and lower sides of the display unit are the scaling boundaries.
A method for synchronous scaling of pictures, comprising:

The mobile terminal receives the zoom command input by the user;

The mobile terminal performs zooming on the displayed original picture according to the zooming command;

If the original picture boundary exceeds the display area boundary, the mobile terminal sends a zoom instruction to the display device, and the zoom instruction includes a display position and a zoom ratio;

The display device moves the position of the target picture according to the display position;

Based on the display position, the target picture is zoomed according to the zoom ratio.