WO2020147246A1 - 同步显示方法及装置、存储介质和电子装置 - Google Patents

同步显示方法及装置、存储介质和电子装置 Download PDF

Info

Publication number
WO2020147246A1
WO2020147246A1 PCT/CN2019/089770 CN2019089770W WO2020147246A1 WO 2020147246 A1 WO2020147246 A1 WO 2020147246A1 CN 2019089770 W CN2019089770 W CN 2019089770W WO 2020147246 A1 WO2020147246 A1 WO 2020147246A1
Authority
WO
WIPO (PCT)
Prior art keywords
terminal
bitmap
screen
rotation state
screenshot information
Prior art date
Application number
PCT/CN2019/089770
Other languages
English (en)
French (fr)
Inventor
黄欣宇
Original Assignee
网易（杭州）网络有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 网易（杭州）网络有限公司 filed Critical 网易（杭州）网络有限公司
Priority to US17/042,938 priority Critical patent/US11340859B2/en
Publication of WO2020147246A1 publication Critical patent/WO2020147246A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • G06F3/1462—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay with means for detecting differences between the image stored in the host and the images displayed on the remote displays
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/39—Control of the bit-mapped memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1423—Digital output to display device ; Cooperation and interconnection of the display device with other functional units controlling a plurality of local displays, e.g. CRT and flat panel display
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/1454—Digital output to display device ; Cooperation and interconnection of the display device with other functional units involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/60—Rotation of whole images or parts thereof
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/36—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
    • G09G5/38—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory with means for controlling the display position
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/04—Changes in size, position or resolution of an image
    • G09G2340/0492—Change of orientation of the displayed image, e.g. upside-down, mirrored

Definitions

  • the present disclosure relates to the computer field, and in particular, to a synchronous display method and device, storage medium and electronic device.
  • the embodiments of the present disclosure provide a synchronous display method and device, storage medium, and electronic device to at least solve the problem that the remote terminal screen needs to be accessed and operated on the PC side during the mobile terminal application development and debugging process of the related technology, but the terminal screen rotates , The synchronization display screen shot is incomplete.
  • a synchronous display method including: acquiring screenshot information of a first terminal, and converting the screenshot information into a corresponding bitmap; acquiring information about the screen of the first terminal Rotation state; in the case where it is determined that the screen is rotated according to the rotation state, a transpose operation corresponding to the rotation state is performed on the bitmap, and the bitmap after the transposition operation is converted into a picture byte stream , And transmit the picture byte stream to the second terminal to synchronously display the screenshot information corresponding to the bitmap; and when it is determined that the screen is not rotated according to the rotation state, the bit The picture is converted into a picture byte stream, and the picture byte stream is transmitted to the second terminal to synchronously display the screenshot information corresponding to the bitmap.
  • a synchronous display device the device includes one or more processors, and one or more memories storing program units, wherein the program units are executed by the processor ,
  • the program unit includes: a first acquisition component, configured to acquire screenshot information of a first terminal, and convert the screenshot information into a corresponding bitmap; and a second acquisition component, configured to acquire the first terminal
  • the rotation state of the screen the display component is configured to perform a transposition operation corresponding to the rotation state on the bitmap when it is determined that the screen is rotated according to the rotation state, and the transpose operation
  • the bitmap is converted into a picture byte stream, and the picture byte stream is transmitted to the second terminal to synchronously display the screenshot information corresponding to the bitmap; and after determining that the screen has not been rotated according to the rotation state
  • the bitmap is converted into a picture byte stream, and the picture byte stream is transmitted to the second terminal to synchronously display the screenshot information corresponding to the bitmap.
  • a storage medium in which a computer program is stored, wherein the computer program is configured to control the device where the storage medium is located to execute any of the foregoing method implementations when running. The steps in the example.
  • an electronic device including a memory and a processor, the memory stores a computer program, the processor is configured to run the computer program to perform any of the above The steps in the method embodiment.
  • a transposition operation corresponding to the rotation state is performed on the bitmap converted from the screenshot information, and the bitmap after the transposition operation is converted into a picture byte stream, And transmit the picture byte stream to the second terminal to synchronously display the screenshot information corresponding to the bitmap, which solves the problem that related technologies need to access and operate the remote terminal screen on the PC side during the development and debugging of mobile terminal applications. If the screen rotates, the synchronously displayed screenshot is incomplete, which improves the synchronous display effect of the remote real machine.
  • FIG. 1 is a hardware structure block diagram of a mobile terminal of a synchronous display method according to one embodiment of the present disclosure
  • Fig. 2 is a flowchart of a synchronous display method according to one of the embodiments of the present disclosure
  • Fig. 3 is a system structure diagram of a real-time on-screen solution for mobile devices according to one of the embodiments of the present disclosure
  • Fig. 4 is a flowchart of a method for executing a mobile device on-screen solution according to one of the embodiments of the present disclosure
  • FIG. 5 is a schematic structural diagram of a synchronous display device according to one of the embodiments of the present disclosure.
  • FIG. 6 is a schematic diagram 1 of an optional structure of a synchronous display device according to one of the embodiments of the present disclosure
  • FIG. 7 is a second schematic diagram of an optional structure of a synchronous display device according to one of the embodiments of the present disclosure.
  • FIG. 8 is a schematic structural diagram of a storage medium according to an embodiment of the present disclosure.
  • FIG. 1 is a hardware structural block diagram of a mobile terminal of a synchronous display method according to one embodiment of the present disclosure.
  • the mobile terminal may include one or more (only one is shown in FIG. 1) processor 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) And the memory 104 for storing data.
  • the above-mentioned mobile terminal may also include a transmission device 106 and an input/output device 108 for communication functions.
  • FIG. 1 is merely an illustration, which does not limit the structure of the mobile terminal described above.
  • the mobile terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration from that shown in FIG.
  • the memory 104 may be used to store computer programs, for example, software programs and components of application software, such as the computer programs corresponding to the synchronous display method in the embodiments of the present disclosure.
  • the processor 102 executes the computer programs stored in the memory 104 by running the computer programs stored in the memory 104. This kind of functional application and data processing realize the above-mentioned method.
  • the memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory.
  • the memory 104 may further include a memory remotely provided with respect to the processor 102, and these remote memories may be connected to the mobile terminal through a network. Examples of the above network include but are not limited to the Internet, intranet, local area network, mobile communication network, and combinations thereof.
  • the transmission device 106 is used to receive or send data via a network.
  • the specific example of the network described above may include a wireless network provided by a communication provider of a mobile terminal.
  • the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through the base station to communicate with the Internet.
  • the transmission device 106 may be a radio frequency (Radio Frequency, RF for short) module, which is used to communicate with the Internet in a wireless manner.
  • RF Radio Frequency
  • FIG. 2 is a flowchart of the synchronous display method according to one embodiment of the present disclosure. As shown in FIG. 2, the process includes the following steps:
  • step S202 obtain screenshot information of the first terminal, and convert the screenshot information into a corresponding bitmap
  • step S204 obtain the rotation state of the screen of the first terminal
  • step S206 when it is determined that the screen is rotated according to the rotation state, a transpose operation corresponding to the rotation state is performed on the bitmap, the bitmap after the transposition operation is converted into a picture byte stream, and the picture bytes Stream to the second terminal to synchronously display the screenshot information corresponding to the bitmap; and when it is determined that the screen is not rotated according to the rotation status, convert the bitmap into a picture byte stream, and transmit the picture byte stream to The second terminal synchronously displays the screenshot information corresponding to the bitmap.
  • a transposition operation corresponding to the rotation state is performed on the bitmap converted from the screenshot information, and the bitmap after the transposition operation It is converted into a picture byte stream, and the picture byte stream is transmitted to the second terminal to synchronously display the screenshot information corresponding to the bitmap, thus solving the problem of the need to access and If the screen of the remote terminal is operated, but the screen of the terminal is rotated, the screen shots of the synchronous display are incomplete, which improves the synchronous display effect of the remote real machine.
  • the execution subject of the above steps S202 to S206 may be a terminal or other devices.
  • the first terminal may be a mobile terminal, such as a mobile phone, a tablet computer, and so on.
  • the second terminal can optionally be a computer terminal or other server equipment.
  • the manner of transposing the bitmap corresponding to the rotation state involved in step S206 can be implemented in the following manner:
  • step S206-11 obtain the return value of the preset rotation function of the window management object, where the return value is used to indicate whether the screen is rotated and the rotation angle;
  • step S206-12 the bitmap is transposed corresponding to the rotation state according to the return value.
  • the return value can indicate that the screen of the first terminal is rotated by 90 degrees when it is equal to 1, and can indicate that the first terminal is rotated when it is equal to 2. If the screen is rotated by 180 degrees, if it is equal to 3, it can indicate that the screen of the first terminal is rotated by 270 degrees, and if it is equal to 0, it can indicate that the screen of the first terminal is not rotated. Therefore, if the return value is equal to 1, the rotation angle of the screen of the first terminal is 90 degrees, and therefore the bitmap needs to be transposed by 90 degrees.
  • the specific value of the return value and the corresponding angle are only examples, and do not constitute a limitation of the present disclosure. In other embodiments, the return value and the value of the corresponding rotation angle may also be set. For example, the return value is equal to 1, and the rotation angle of the screen of the first terminal is 270.
  • the method of this embodiment may further include:
  • step S102 trigger the second terminal to start the first thread
  • step S104 a socket connection between the first terminal and the second terminal is established based on the first thread.
  • the first terminal becomes a socket server.
  • the method of transmitting the bitmap after the transposition operation to the second terminal involved in step S206 in this embodiment to synchronously display the screenshot information corresponding to the bitmap may further include:
  • step S206-13 the bitmap after the transposition operation is converted into a byte stream in a picture format, and the byte stream is encapsulated;
  • step S206-14 the encapsulated byte stream is transmitted to the second terminal through the socket connection to synchronously display the screenshot information corresponding to the byte stream.
  • the byte stream converted into the adobe format can be transmitted to the second terminal through the socket connection, and the screenshot information corresponding to the byte stream can be displayed synchronously.
  • the method of this embodiment may further include:
  • step S106 before acquiring the screenshot information of the first terminal, trigger the second terminal to start a second thread, where the second thread is used to restart the acquisition of the first terminal when the screenshot information of the first terminal fails to be acquired.
  • the operation of the screenshot information of the terminal before acquiring the screenshot information of the first terminal, trigger the second terminal to start a second thread, where the second thread is used to restart the acquisition of the first terminal when the screenshot information of the first terminal fails to be acquired.
  • the second thread can ensure that the operation of obtaining the screenshot information of the first terminal can be successfully completed, and the synchronization display operation will not be terminated after the operation fails.
  • Accessing and operating the remote mobile phone screen on the PC side can be used to solve the problem of inefficiency caused by the frequent need to connect the PC to the mobile phone and switch back and forth between the mobile phone and the PC screen during the development and debugging of Android mobile phone applications.
  • the debugged mobile phone screen can be displayed on the PC screen in real time through certain technical means.
  • minicap uses the minicap executable file on the mobile phone to start a socket server to accept the connection from the PC client.
  • Minicap continuously takes screenshots through the ndk interface, and the screen frames are continuously sent to the socket client on the PC side. (socket client).
  • the PC side implements socket connection forwarding through the adb forward command.
  • the socket client of the PC connects to the socket server opened by the minicap through adb forward forwarding, and reads the frame stream continuously sent by the mobile phone, so that the PC can get the sequence of picture streams, which can then be processed on the PC through the graphical user interface ( Graphical User Interface (referred to as GUI) program or Web page display.
  • GUI Graphical User Interface
  • this specific embodiment takes the first terminal as an Android mobile phone and the second terminal as a PC as an example, and provides a way to improve the compatibility and stability of screenshots by calling the native java API interface of the Android bottom layer, and The logical control solves the problem of screen rotation and black screen, which can be effectively used to optimize the same-screen display method of the remote real machine.
  • the PC side implements socket connection forwarding through the adb forward command.
  • the socket client of the PC connects to the socket server of the mobile phone opened by the app_process through adb forward, so that the transmitted Android screen can be read in real time. After the PC is processed, it can be displayed on the same screen in real time through GUI programs or Web pages.
  • FIG. 3 is a system structure diagram of a real-time on-screen solution for mobile devices according to one embodiment of the present disclosure. As shown in Figure 3, the system includes: a front-end web page, a server process (Yosemite process) and a client agent module (Agent).
  • a server process Yosemite process
  • Agent client agent module
  • the front-end webpage is used to establish a socket connection (websocket) connection with the screenshot interface of the server, so as to continuously obtain screenshots of the remote real machine and display them on the front-end webpage.
  • a socket connection websocket
  • the Yosemite process on the server side includes two parts: a javacap thread (equivalent to the aforementioned first thread) and a monitoring thread (equivalent to the aforementioned second thread).
  • the monitoring thread is used to start the Agent module in the mobile phone client (App) and monitor the status of the Agent module. When the Agent module hangs up on the mobile phone, it will restart to ensure stable screenshots.
  • the Yosemite process will start the javacap thread.
  • the javacap thread establishes a socket connection with the Agent. By establishing the socket connection, the binary stream of real-time screenshots of the Agent module can be obtained for the front-end page to obtain.
  • the Agent module runs on a remote real machine and is started and monitored by the monitor thread of the server Yosemite.
  • the Agent module connects with Yosemite's javacap thread through Socket, and can communicate through protobuf protocol.
  • the Agent will continue to call the underlying interface (screenshot) to obtain the Bitmap of the screenshot, and convert the Bitmap into a jpg format image binary stream, and then encapsulate it into the format specified by the protocol (protobuf) and send it To the javacap thread for processing.
  • Fig. 4 is a flowchart of a method for executing a mobile device on-screen solution according to one of the embodiments of the present disclosure. As shown in Fig. 4, the steps of the method include:
  • step S401 the Agent process (agent module) on the remote mobile phone is started through the app_process command of adb.
  • step S402 after the agent process (Agent process) is started on the mobile phone, the Socket thread is started, so that the mobile phone becomes a socket server.
  • agent process Agent process
  • step S403 waiting for the socket connection, the Socket thread keeps waiting for the connection from the server Yosemite.
  • step S404 the server will start the first process (javacap thread) for establishing a socket connection with the remote device, and cyclically receiving the remote device screenshot byte stream sent by the Agent.
  • step S405 the port of the server can be redirected to the port of the socket server of the mobile phone through the forward instruction of adb, and a Socket connection can be established.
  • step S406 after the Socket thread waits for the connection from the server-side Yosemite, it will determine whether the socket connection is successful. If it is determined that the socket connection is successful, perform step S407. If it is determined that the socket connection is unsuccessful, Step S403 is executed.
  • step S407 call the underlying interface (screenshot), and the screenshot thread will establish a loop;
  • step S408 after the screenshot() method is called, a Bitmap object corresponding to the picture is generated.
  • step S409 the getRotation method of IWindowManager can be used to determine whether the screen is rotated. If it is determined that the screen is rotated, step S410 is executed, and if it is determined that the screen is not rotated, step S411 is executed.
  • step S410 the return value of the preset rotation function of the window management object, if the return value is equal to 1, a 90 degree rotation will occur, if the return value is equal to 2, a 180 degree rotation will occur, and if the return value is equal to 3, a 270 degree rotation will occur, If the return value is equal to 0, no rotation has occurred.
  • step S410 if rotation occurs, the Bitmap is transposed by the corresponding degree of rotation.
  • step S411 if no rotation occurs, the compress method of the Bitmap class is called to convert the Bitmap into a byte stream of the corresponding picture format, and can be compressed into a picture byte stream of different formats and definitions through the compress method.
  • step S412 after being converted into a byte stream, the agent transmits the byte stream of the intercepted picture back to the javacap thread via TCP.
  • the solution of this specific implementation mode can support most Android system smart phones through the interface
  • the form of invoking the start service can improve the compatibility of the screenshot device.
  • the screenshot scheme of the Android system in the related technology if the screen is rotated, the screenshot often has a black screen and black edges.
  • the current screen rotation state is determined immediately after calling the screenshot interface to obtain the Bitmap. If rotation occurs, the Bitmap is immediately transposed to ensure that when the Bitmap is converted into a picture, the picture can be displayed normally.
  • the PC in this specific embodiment will monitor the running status of the client service process, and if an abnormal exit occurs during the screenshot capture process, the client process can be restarted to ensure the smooth progress of subsequent data operations.
  • the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is Better implementation.
  • the technical solution of the present disclosure can be embodied in the form of a software product in essence or part that contributes to the existing technology, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk,
  • the CD-ROM includes several instructions to enable a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to perform the methods described in the embodiments of the present disclosure.
  • a synchronous display device includes one or more processors, and one or more memories storing program units, where the program units are executed by the processors, and the program units may include first The acquisition component, the second acquisition component, and the display component.
  • the device is used to implement the above-mentioned embodiments and preferred implementations, and those that have been described will not be repeated.
  • the term “component” may be a combination of software and/or hardware that implements predetermined functions.
  • FIG. 5 is a schematic structural diagram of a synchronous display device according to one of the embodiments of the present disclosure.
  • the device includes: a first obtaining component 502, which is configured to obtain screenshot information of the first terminal and take the screenshot The information is converted into a corresponding bitmap; the second acquisition component 504, coupled to the first acquisition component 502, is configured to acquire the rotation state of the screen of the first terminal; the display component 506, coupled to the second acquisition component 504, is Set to perform the transposition operation corresponding to the rotation state on the bitmap when the screen is rotated according to the rotation state, convert the transposed bitmap into a picture byte stream, and transmit the picture byte stream to The second terminal synchronously displays the screenshot information corresponding to the bitmap; and when it is determined that the screen does not rotate according to the rotation state, the bitmap is converted into a picture byte stream, and the picture byte stream is transmitted to the second terminal To display the screenshot information corresponding to the bitmap synchronously.
  • the first acquiring component 502, the second acquiring component 504, and the display component 506 can be run in the terminal as a part of the device, and the functions implemented by the aforementioned components can be executed by the processor in the terminal. It can be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, an applause computer, and a mobile Internet device (Mobile Internet Devices, referred to as MID), PAD and other terminal devices.
  • MID Mobile Internet Devices
  • the display component 506 involved in this embodiment may further include: an acquisition element configured to acquire the return value of the preset rotation function of the window management object, where the return value is used to indicate whether the screen is rotated and the rotation angle;
  • the transpose component is set to perform the transpose operation corresponding to the rotation state on the bitmap according to the return value.
  • the aforementioned acquisition element and transposition element may be run in a terminal as a part of the device, and the functions implemented by the aforementioned elements may be executed by a processor in the terminal.
  • Fig. 6 is a schematic diagram 1 of an optional structure of a synchronous display device according to one of the embodiments of the present disclosure.
  • the device may further include: a first triggering component 602 coupled to the first acquiring component 502 and configured To trigger the second terminal to start the first thread before acquiring the screenshot information of the first terminal; the establishment component 604 is coupled to the first trigger component 602, and is set to establish the relationship between the first terminal and the second terminal based on the first thread. Socket connection between.
  • first triggering component 602 and establishment component 604 may be run in the terminal as a part of the device, and the functions implemented by the above-mentioned components may be executed by the processor in the terminal.
  • FIG. 6 is a schematic diagram 1 of an optional structure of a synchronous display device according to one of the embodiments of the present disclosure
  • the display component 506 involved in this embodiment may also include: a processing element, which is configured to convert the transposed bitmap into a byte stream in a picture format, and compare the byte The stream is encapsulated; the display element is set to transmit the encapsulated byte stream to the second terminal through a socket connection to synchronously display the screenshot information corresponding to the byte stream.
  • a processing element which is configured to convert the transposed bitmap into a byte stream in a picture format, and compare the byte The stream is encapsulated
  • the display element is set to transmit the encapsulated byte stream to the second terminal through a socket connection to synchronously display the screenshot information corresponding to the byte stream.
  • processing element and display element may be run in the terminal as part of the device, and the functions implemented by the foregoing element may be executed by the processor in the terminal.
  • FIG. 7 is a second schematic diagram of an optional structure of a synchronous display device according to one of the embodiments of the present disclosure.
  • the device further includes: a second triggering component 702 coupled to the first acquiring component 502 and configured to Before acquiring the screenshot information of the first terminal, trigger the second terminal to start a second thread, where the second thread is used to restart the acquisition of the screenshot information of the first terminal in the case that the acquisition of the screenshot information of the first terminal fails Operation.
  • the above-mentioned second trigger component 702 may be run in a terminal as a part of an apparatus, and the functions implemented by the above-mentioned components may be executed by a processor in the terminal.
  • the above components can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but not limited to this: the above components are all located in the same processor; or, the above components can be combined in any combination The forms are located in different processors.
  • the embodiment of the present disclosure also provides a storage medium in which a computer program is stored, wherein the computer program is configured to control the device where the storage medium is located to execute the steps in any one of the foregoing method embodiments when running.
  • FIG. 8 is a schematic structural diagram of a storage medium according to an embodiment of the present disclosure. As shown in FIG. 8, a program product 800 according to an embodiment of the present disclosure is described, and a computer program is stored thereon. When the computer program is executed by a processor, the program code realizes the following steps:
  • S1 Obtain screenshot information of the first terminal, and convert the screenshot information into a corresponding bitmap
  • the program code when executed by the processor, the program code also implements the following steps:
  • the storage medium is also configured to store a computer program for executing the following steps:
  • S2 Establish a socket connection between the first terminal and the second terminal based on the first thread.
  • the program code when executed by the processor, the program code also implements the following steps:
  • S1 Before acquiring the screenshot information of the first terminal, trigger the second terminal to start a second thread, where the second thread is used to restart if the acquisition of the screenshot information of the first terminal fails The operation of obtaining screenshot information of the first terminal.
  • a transposition operation corresponding to the rotation state is performed on the bitmap converted from the screenshot information, and the transposed bitmap is converted into a picture byte stream , And transmit the picture byte stream to the second terminal to synchronously display the screenshot information corresponding to the bitmap, thereby solving the problem of the need to access and operate the remote terminal screen on the PC side during the development and debugging of mobile terminal applications. If the terminal screen rotates, the screen shots displayed simultaneously are incomplete, which improves the synchronization display effect of the remote real machine.
  • the computer-readable storage medium may include a data signal propagated in baseband or as a part of a carrier wave, and readable program code is carried therein.
  • This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable storage medium can send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device.
  • the program code contained in the computer-readable storage medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, radio frequency, etc., or any suitable combination of the foregoing.
  • the above storage medium may include, but is not limited to: a USB flash drive, a read-only memory (Read-Only Memory, ROM for short), a random access memory (Random Access Memory, RAM for short), Various media that can store computer programs, such as removable hard disks, magnetic disks, or optical disks.
  • the embodiment of the present disclosure further provides an electronic device, including a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above method embodiments , Can be a mobile terminal as shown in Figure 1.
  • the electronic device may further include a transmission device and an input-output device, where the transmission device is connected to the processor, and the input-output device is connected to the processor.
  • the foregoing processor may be configured to perform the following steps through a computer program:
  • S1 Obtain screenshot information of the first terminal, and convert the screenshot information into a corresponding bitmap
  • the processor is also configured to store a computer program for performing the following steps:
  • the storage medium is also configured to store a computer program for executing the following steps:
  • S2 Establish a socket connection between the first terminal and the second terminal based on the first thread.
  • the processor is also configured to store a computer program for performing the following steps:
  • S1 Before acquiring the screenshot information of the first terminal, trigger the second terminal to start a second thread, where the second thread is used to restart if the acquisition of the screenshot information of the first terminal fails The operation of obtaining screenshot information of the first terminal.
  • a transposition operation corresponding to the rotation state is performed on the bitmap converted from the screenshot information, and the transposed bitmap is converted into a picture byte stream , And transmit the picture byte stream to the second terminal to synchronously display the screenshot information corresponding to the bitmap, thereby solving the problem of the need to access and operate the remote terminal screen on the PC side during the development and debugging of mobile terminal applications. If the terminal screen rotates, the screen shots displayed simultaneously are incomplete, which improves the synchronization display effect of the remote real machine.
  • the foregoing processor can execute various functional applications and data processing by running software programs and components stored in the memory, that is, realize the foregoing synchronous display method.
  • a transposition operation corresponding to the rotation state is performed on the bitmap converted from the screenshot information, and the bitmap after the transposition operation is converted into a picture byte stream, And transmit the picture byte stream to the second terminal to synchronously display the screenshot information corresponding to the bitmap, which solves the problem that related technologies need to access and operate the remote terminal screen on the PC side during the development and debugging of mobile terminal applications. If the screen rotates, the synchronously displayed screenshot is incomplete, which improves the synchronous display effect of the remote real machine.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • User Interface Of Digital Computer (AREA)
  • Debugging And Monitoring (AREA)
  • Digital Computer Display Output (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

一种同步显示方法及装置、存储介质和电子装置，其中该方法包括：获取第一终端的屏幕截图信息，并将屏幕截图信息转化为对应的位图(S202)；获取第一终端的屏幕的旋转状态(S204)；在根据旋转状态确定屏幕发生旋转的情况下，对位图执行与旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将图片字节流传输到第二终端以同步显示与位图对应的屏幕截图信息。

Description

同步显示方法及装置、存储介质和电子装置
本申请要求于2019年01月17日提交中国专利局、优先权号为201910045548.0、发明名称为“同步显示方法及装置、存储介质和电子装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本公开涉及计算机领域，具体而言，涉及一种同步显示方法及装置、存储介质和电子装置。
背景技术
目前，在安卓(Android)手机应用的开发调试过程中，需要在个人计算机(Personal Compute，简称为PC)端访问和操作远程终端屏幕，通常采用的是基于开源代码库(Github)的远程操作安卓手机工具(Smartphone Test Farm，简称为STF)工具的框架(minicap)，利用minicap可以实现手机的实时展示。
但是，在使用过程中minicap存在以下问题：在截图过程中，如果屏幕发生旋转，则收到的截图有黑色区域，使得图片显示不完整。
针对相关技术中的上述问题，目前尚未存在有效的解决方案。
发明内容
本公开实施例提供了一种同步显示方法及装置、存储介质和电子装置，以至少解决相关技术在移动终端应用开发调试过程中，需要在PC端访问和操作远程终端屏幕，但是终端屏幕发生旋转，则同步显示的屏幕截图不完整的问题。
根据本公开的一个实施例，提供了一种同步显示方法，包括：获取第一终端的屏幕截图信息，并将所述屏幕截图信息转化为对应的位图；获取所述第一终端的屏幕的旋转状态；在根据所述旋转状态确定所述屏幕发生旋转的情况下，对所述位图执行与所述旋转状态对应的转置操作，将转置 操作后的位图转化为图片字节流，并将所述图片字节流传输到第二终端以同步显示与所述位图对应的屏幕截图信息；以及在根据所述旋转状态确定所述屏幕未发生旋转的情况下，将所述位图转化为图片字节流，并将所述图片字节流传输到所述第二终端以同步显示与所述位图对应的屏幕截图信息。
根据本公开的另一个方面，提供了一种同步显示装置，所述装置包括一个或多个处理器，以及一个或多个存储程序单元的存储器，其中，所述程序单元由所述处理器执行，所述程序单元包括：第一获取组件，设置为获取第一终端的屏幕截图信息，并将所述屏幕截图信息转化为对应的位图；第二获取组件，设置为获取所述第一终端的屏幕的旋转状态；显示组件，设置为在根据所述旋转状态确定所述屏幕发生旋转的情况下，对所述位图执行与所述旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将所述图片字节流传输到第二终端以同步显示与所述位图对应的屏幕截图信息；以及在根据所述旋转状态确定所述屏幕未发生旋转的情况下，将所述位图转化为图片字节流，并将所述图片字节流传输到第二终端以同步显示与所述位图对应的屏幕截图信息。
根据本公开的又一个实施例，还提供了一种存储介质，所述存储介质中存储有计算机程序，其中，所述计算机程序被设置为运行时控制存储介质所在设备执行上述任一项方法实施例中的步骤。
根据本公开的又一个实施例，还提供了一种电子装置，包括存储器和处理器，所述存储器中存储有计算机程序，所述处理器被设置为运行所述计算机程序以执行上述任一项方法实施例中的步骤。
通过本公开，在根据旋转状态确定屏幕发生旋转的情况下，对由屏幕截图信息转化的位图执行与旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将图片字节流传输到第二终端以同步显示与位图对应的屏幕截图信息，从而解决了相关技术在移动终端应用开发调试过程中，需要在PC端访问和操作远程终端屏幕，但是终端屏幕发生旋转，则 同步显示的屏幕截图不完整的问题，达到了提高远程真机的同步显示效果。
附图说明
此处所说明的附图用来提供对本公开实施例的进一步理解，构成本申请的一部分，本公开的示意性实施例及其说明用于解释本公开，并不构成对本公开的不当限定。在附图中：
图1是本公开其中一实施例的一种同步显示方法的移动终端的硬件结构框图；
图2是根据本公开其中一实施例的同步显示方法的流程图；
图3是根据本公开其中一实施例的移动设备实时同屏方案的系统结构图；
图4是根据本公开其中一实施例的执行移动设备同屏方案的方法流程图；
图5是根据本公开其中一实施例的同步显示装置的结构示意图；
图6是根据本公开其中一实施例的同步显示装置的可选结构示意图一；
图7是根据本公开其中一实施例的同步显示装置的可选结构示意图二；
图8是根据本公开其中一实施例的存储介质的结构示意图。
具体实施方式
下文中将参考附图并结合实施例来详细说明本公开。需要说明的是，在不冲突的情况下，本申请中的实施例及实施例中的特征可以相互组合。
本申请其中之一实施例所提供的方法实施例可以在计算机终端或者类似的运算装置中执行。以运行在移动终端上为例，图1是本公开其中一实施例的同步显示方法的移动终端的硬件结构框图。如图1所示，移动终端可以包括一个或多个(图1中仅示出一个)处理器102(处理器102可以包括但不限于微处理器MCU或可编程逻辑器件FPGA等的处理装置)和用于存储数据的存储器104，可选地，上述移动终端还可以包括用于通 信功能的传输设备106以及输入输出设备108。本领域普通技术人员可以理解，图1所示的结构仅为示意，其并不对上述移动终端的结构造成限定。例如，移动终端还可包括比图1中所示更多或者更少的组件，或者具有与图1所示不同的配置。
存储器104可用于存储计算机程序，例如，应用软件的软件程序以及组件，如本公开实施例中的同步显示方法对应的计算机程序，处理器102通过运行存储在存储器104内的计算机程序，从而执行各种功能应用以及数据处理，即实现上述的方法。存储器104可包括高速随机存储器，还可包括非易失性存储器，如一个或者多个磁性存储装置、闪存、或者其他非易失性固态存储器。在一些实例中，存储器104可进一步包括相对于处理器102远程设置的存储器，这些远程存储器可以通过网络连接至移动终端。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
传输设备106用于经由一个网络接收或者发送数据。上述的网络具体实例可包括移动终端的通信供应商提供的无线网络。在一个实例中，传输设备106包括一个网络适配器(Network Interface Controller，简称为NIC)，其可通过基站与其他网络设备相连从而可与互联网进行通讯。在一个实例中，传输设备106可以为射频(Radio Frequency，简称为RF)模块，其用于通过无线方式与互联网进行通讯。
在本实施例中提供了一种运行于上述移动终端的同步显示方法，图2是根据本公开其中一实施例的同步显示方法的流程图，如图2所示，该流程包括如下步骤：
在步骤S202中，获取第一终端的屏幕截图信息，并将屏幕截图信息转化为对应的位图；
在步骤S204中，获取第一终端的屏幕的旋转状态；
在步骤S206中，在根据旋转状态确定屏幕发生旋转的情况下，对位图执行与旋转状态对应的转置操作，将转置操作后的位图转化为图片字节 流，并将图片字节流传输到第二终端以同步显示与位图对应的屏幕截图信息；以及在根据旋转状态确定屏幕未发生旋转的情况下，将位图转化为图片字节流，并将图片字节流传输到第二终端以同步显示与位图对应的屏幕截图信息。
通过上述步骤S202至步骤S206，在根据旋转状态确定屏幕发生旋转的情况下，对由屏幕截图信息转化的位图(Bitmap)执行与旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将图片字节流传输到第二终端以同步显示与位图对应的屏幕截图信息，从而解决了相关技术在移动终端应用开发调试过程中，需要在PC端访问和操作远程终端屏幕，但是终端屏幕发生旋转，则同步显示的屏幕截图不完整的问题，达到了提高远程真机的同步显示效果。
需要说明的是，上述步骤S202至步骤S206的执行主体可以是终端，或其他设备。此外，第一终端可选为移动终端，例如：手机、平板电脑等等。第二终端可选为计算机终端或其他服务器设备。
在本实施例的可选实施方式中，对于步骤S206中涉及的在确定屏幕发生旋转的情况下，对位图进行与旋转状态对应的转置操作的方式，可以通过如下方式来实现：
在步骤S206-11中，获取视窗管理对象的预设旋转函数的返回值，其中，返回值用于指示屏幕是否旋转以及旋转角度；
在步骤S206-12中，按照返回值对位图执行与旋转状态对应的转置操作。
对于上述步骤S206-11和步骤S206-12，在具体应用场景中，该返回值在等于1的情况下可以指示第一终端的屏幕发生90度旋转，在等于2的情况下可以指示第一终端的屏幕发生180度旋转，在等于3的情况下可以指示第一终端的屏幕发生270度旋转，在等于0的情况下可以指示第一终端的屏幕没有发生旋转。因此，如果返回值等于1，则第一终端的屏幕的旋转角度为90度，因此则需要对该位图进行90度旋转的转置操作。当 然，上述返回值的具体取值以及对应的角度仅仅是举例说明，并不构成对本公开的限定。在其他实施方式中也可以设置返回值与对应旋转角度的值，例如返回值等于1，第一终端的屏幕的旋转角度为270。
在本实施例的另一个可选实施方式中，在步骤S202获取第一终端的屏幕截图信息之前，本实施例的方法还可以包括：
在步骤S102中，触发第二终端启动第一线程；
在步骤S104中，基于第一线程建立第一终端与第二终端之间的套接字连接。
基于上述步骤S102和步骤S104可知，在基于第一线程建立第一终端与第二终端之间的套接字连接之后，该第一终端就变成了套接字服务器(socket server)。
基于上述套接字连接，对于本实施例中步骤S206中涉及的将进行转置操作后的位图传输到第二终端以同步显示与位图对应的屏幕截图信息的方式，可以进一步包括：
在步骤S206-13中，将进行转置操作后的位图转化成图片格式的字节流，并对字节流进行封装；
在步骤S206-14中，通过套接字连接将封装后的字节流传输到第二终端以同步显示与字节流对应的屏幕截图信息。
通过上述步骤S206-13和步骤S206-14可知，通过套接字连接可以将转化成土坯格式的字节流传输到第二终端，进而同步显示与字节流对应的屏幕截图信息。
在本实施例的再一个可选实施方式中，本实施例的方法还可以包括：
在步骤S106中，在获取第一终端的屏幕截图信息之前，触发第二终端启动第二线程，其中，第二线程用于在获取第一终端的屏幕截图信息失败的情况下，重启获取第一终端的屏幕截图信息的操作。
可见，通过该第二线程能够保证获取第一终端的屏幕截图信息的操作 能够顺利完成，不会因为该操作失败后导致同步显示操作终止。
下面结合本实施例的具体实施方式来对本申请进行举例说明：
在PC端访问和操作远程手机屏幕，可以用于解决在Android手机应用开发调试过程中，经常需要PC连接手机，来回手机和PC屏幕切换导致的效率低下的问题。在上述情况下，可以通过一定技术手段将调试的手机屏幕在PC屏幕上实时显示。
在Android手机应用开发调试过程中，相关技术中常采用的是基于Github的手机测试项目STF的子项目minicap，利用minicap可以实现手机的实时展示。
minicap的基本原理为：在手机端通过minicap可执行文件，启动一个socket server，接受来自PC的客户端的连接，minicap通过ndk接口不停的截屏，屏幕帧不断的发送到PC端的套接字客户端(socket client)上。PC端通过adb forward命令实现socket连接转发。PC的socket client通过adb forward转发连接至minicap开启的socket server，读取手机端不断发送的帧流，这样PC端便可以得到序列的图片流，然后在PC端处理之后即可通过图形用户界面(Graphical User Interface，简称为GUI)程序或者Web页面展示。
但是，在截图过程中，如果屏幕发生旋转，收到的截图有黑色区域，图片显示不完整。minicap经过测试，目前只支持mtl中70％左右的机型，兼容性不够理想。并且由于获取不到截图，经常会引起系统崩溃。
为了解决上述问题，本具体实施例以第一终端为Android手机、第二终端为PC端为例，提供了一种通过调用Android底层原生java API接口的方式以提升截图兼容性和稳定性，并通过逻辑控制解决了屏幕旋转黑屏问题，可以有效地用于优化远程真机的同屏显示方法。
首先，对本具体实施方式的核心思想进行以下几点说明：
(1)在本具体实施方式中，借助Java底层反射原理，调用Android底层的接口screenshot()方法，通过adb shell的app_process可以解决隐藏 接口SurfaceControl.screenshot()/Surface.screenshot()方法的调用权限问题。该方法调用返回结果是个Bitmap，然后通过相应的字节流转化，就能够顺利获取手机屏幕截图。
(2)调用IWindowManager的getRotation方法获取当前屏幕是否翻转。根据返回结果调用Matrix类的postRotate方法可以对Bitmap做相应的翻转，从而实现图片旋转的截屏。
(3)实现一个Android应用程序，安装在远程手机中。在应用程序中实现调用SurfaceControl.screenshot()/Surface.screenshot的方法。将手机通过连接线连接PC，在PC端通过指令(app_process)启动应用程序，这样手机就变成了一台socket server。
(4)PC端通过adb forward命令实现socket连接转发。PC的socket client通过adb forward转发连接至app_process开启的手机端socket server，这样就可以实时读取传输的Android屏幕。PC端经过处理之后即可通过GUI程序或者Web页面实时同屏显示。
基于上述(1)～(4)，本具体实施方式提供了一种移动设备实时同屏方案的系统，图3是根据本公开其中一实施例的移动设备实时同屏方案的系统结构图。如图3所示，该系统包括：前端网页，服务器进程(Yosemite进程)以及客户端的代理模块(Agent)。
其中，前端网页用于与服务器的截图接口建立套接字连接(websocket)连接，从而不断的获取远程真机的截屏图片，在前端网页中显示。
服务端Yosemite进程包括：javacap线程(相当于上述的第一线程)和监控线程(相当于上述的第二线程)两部分。其中，监控线程用于启动手机客户端(App)中的Agent模块，并对Agent模块的状态进行监控，当Agent模块在手机端挂掉时，会进行重启操作，保证截图稳定进行。
在Agent启动之后，Yosemite进程会启动javacap线程，javacap线程同Agent建立socket连接，通过建立socket连接，可以获取Agent模块实时截图的二进制流，可供前端页面进行获取。
需要说明的是，Agent模块运行在远程真机上，通过服务器Yosemite的monitor线程进行启动和监控。Agent模块通过Socket同Yosemite的javacap线程进行连接，可以通过protobuf协议进行通讯。在Agent同javacap建立socket连接后，Agent就会不断调用底层接口(screenshot)，获取截图的Bitmap，并将Bitmap转化成jpg格式的图片二进制流，再将其封装成协议(protobuf)规定的格式发送给javacap线程进行处理。
图4是根据本公开其中一实施例的执行移动设备同屏方案的方法流程图，如图4所示，该方法的步骤包括：
在步骤S401中，通过adb的app_process命令启动远程手机上的Agent进程(代理模块)。
在步骤S402中，在代理进行(Agent进程)在手机端启动后，就会开启Socket线程，从而手机变成了一台套接字服务器(socket server)。
在步骤S403中，等待套接字连接，Socket线程不断等待来自服务器端Yosemite的连接。
在步骤S404中，服务器端会启动第一进程(javacap线程)，用于建立同远程设备的socket连接，以及循环接收Agent发送的远程设备截图字节流。
在步骤S405中，通过adb的forward指令可以将服务器端的端口重定向到手机端的socket server的端口上，并建立Socket连接。
在步骤S406中，在Socket线程等待来自服务器端Yosemite的连接之后，就会判断套接字连接是否成功，如果判断出套接字连接成功，执行步骤S407，如果判断出套接字连接不成功，执行步骤S403。
在步骤S407中，调用底层接口(screenshot)，屏幕截图线程会建立一个循环；
其中，在Android 4.2及以下版本线程会调用Surface.screenshot()方法，在Android 4.2以上版本则调用SurfaceControl.screenshot()方法。
在步骤S408中，在调用screenshot()方法之后，就会生成一个图片对应的Bitmap对象。
在步骤S409中，通过IWindowManager的getRotation方法可以判断屏幕是否发生旋转。如果判断出屏幕发生旋转，执行步骤S410，如果判断出屏幕没有发生旋转，执行步骤S411。可选地，获取视窗管理对象的预设旋转函数的返回值，如果返回值等于1则发生90度旋转，如果返回值等于2则发生180度旋转，如果返回值等于3则发生270度旋转，如果返回值等于0则没有发生旋转。
在步骤S410中，如果发生旋转，则对Bitmap做相应旋转度的转置操作。
在步骤S411中，如果没有发生旋转，则调用Bitmap类的compress方法，可以将Bitmap转化成相应图片格式的字节流，可以通过compress方法压缩成不同的格式和清晰度的图片字节流。
在步骤S412中，在转化成字节流后，Agent将截取图片的字节流通过TCP传输回给javacap线程。
通过本具体实施方式的上述方法步骤，能够带来以下有益效果：
由于Android系统底层都支持隐藏接口SurfaceControl.screenshot()/Surface.screenshot()，并可以使用root获取接口的调用权限，因此，本具体实施方式的方案可以支持大多数Android系统的智能手机，通过接口调用启动服务的形式，从而可以提高截图设备的兼容性。此外，相关技术中安卓系统截图方案，如果屏幕旋转，则截图经常出现黑屏和黑边。通过本具体实施方式在调用screenshot接口获取Bitmap后立即判断屏幕当前旋转状态，若出现旋转，则立即对Bitmap做出相应的转置处理，从而保证Bitmap转化成图片时，图片能够正常显示。最后，本具体实施方式中的PC端会监控客户端服务进程的运行状态，如果在获取截图过程中出现异常退出，能够重启客户端进程，保证后续数据操作顺利进行。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到根据上述实施例的方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，或者网络设备等)执行本公开各个实施例所述的方法。
在本实施例中还提供了一种同步显示装置，该装置包括一个或多个处理器，以及一个或多个存储程序单元的存储器，其中，程序单元由处理器执行，程序单元可以包括第一获取组件、第二获取组件和显示组件。该装置用于实现上述实施例及优选实施方式，已经进行过说明的不再赘述。如以下所使用的，术语“组件”可以实现预定功能的软件和/或硬件的组合。尽管以下实施例所描述的装置较佳地以软件来实现，但是硬件，或者软件和硬件的组合的实现也是可能并被构想的。
图5是根据本公开其中一实施例的同步显示装置的结构示意图，如图5所示，该装置包括：第一获取组件502，被设置为获取第一终端的屏幕截图信息，并将屏幕截图信息转化为对应的位图；第二获取组件504，与第一获取组件502耦合连接，被设置为获取第一终端的屏幕的旋转状态；显示组件506，与第二获取组件504耦合连接，被设置为在根据旋转状态确定屏幕发生旋转的情况下，对位图执行与旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将图片字节流传输到第二终端以同步显示与位图对应的屏幕截图信息；以及在根据旋转状态确定屏幕未发生旋转的情况下，将位图转化为图片字节流，并将图片字节流传输到第二终端以同步显示与位图对应的屏幕截图信息。
此处需要说明的是，上述第一获取组件502、第二获取组件504和显示组件506可以作为装置的一部分运行在终端中，可以通过终端中的处理器来执行上述组件实现的功能，终端也可以是智能手机(如Android手机、 iOS手机等)、平板电脑、掌声电脑以及移动互联网设备(Mobile Internet Devices，简称为MID)、PAD等终端设备。
可选地，本实施例中涉及的显示组件506还可以包括：获取元件，被设置为获取视窗管理对象的预设旋转函数的返回值，其中，返回值用于指示屏幕是否旋转以及旋转角度；转置元件，被设置为按照返回值对位图执行与旋转状态对应的转置操作。
此处需要说明的是，上述获取元件和转置元件可以作为装置的一部分运行在终端中，可以通过终端中的处理器来执行上述元件实现的功能。
图6是根据本公开其中一实施例的同步显示装置的可选结构示意图一，如图6所示，该装置还可以包括：第一触发组件602，与第一获取组件502耦合连接，被设置为在获取第一终端的屏幕截图信息之前，触发第二终端启动第一线程；建立组件604，与第一触发组件602耦合连接，被设置为基于第一线程建立第一终端与第二终端之间的套接字连接。
此处需要说明的是，上述第一触发组件602和建立组件604可以作为装置的一部分运行在终端中，可以通过终端中的处理器来执行上述组件实现的功能。
图6是根据本公开其中一实施例的同步显示装置的可选结构示意图一，
基于图6中该装置的组件，本实施例中涉及的显示组件506还可以包括：处理元件，被设置为将进行转置操作后的位图转化成图片格式的字节流，并对字节流进行封装；显示元件，被设置为通过套接字连接将封装后的字节流传输到第二终端以同步显示与字节流对应的屏幕截图信息。
此处需要说明的是，上述处理元件和显示元件可以作为装置的一部分运行在终端中，可以通过终端中的处理器来执行上述元件实现的功能。
图7是根据本公开其中一实施例的同步显示装置的可选结构示意图二，如图7所示，该装置还包括：第二触发组件702，与第一获取组件502耦合连接，被设置为在获取第一终端的屏幕截图信息之前，触发第二终端启动第二线程，其中，第二线程用于在获取第一终端的屏幕截图信息失败的 情况下，重启获取第一终端的屏幕截图信息的操作。
上述此处需要说明的是，上述第二触发组件702可以作为装置的一部分运行在终端中，可以通过终端中的处理器来执行上述组件实现的功能。
需要说明的是，上述各个组件是可以通过软件或硬件来实现的，对于后者，可以通过以下方式实现，但不限于此：上述组件均位于同一处理器中；或者，上述各个组件以任意组合的形式分别位于不同的处理器中。
本公开的实施例还提供了一种存储介质，该存储介质中存储有计算机程序，其中，该计算机程序被设置为运行时控制存储介质所在设备执行上述任一项方法实施例中的步骤。
图8是根据本公开其中一实施例的存储介质的结构示意图。如图8所示，描述了根据本公开的实施方式的程序产品800，其上存储有计算机程序，计算机程序被处理器执行时实现如下步骤的程序代码：
S1，获取第一终端的屏幕截图信息，并将所述屏幕截图信息转化为对应的位图；
S2，获取所述第一终端的屏幕的旋转状态；
S3，在根据所述旋转状态确定所述屏幕发生旋转的情况下，对所述位图执行与所述旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将所述图片字节流传输到第二终端以同步显示与所述位图对应的屏幕截图信息；以及在根据所述旋转状态确定所述屏幕未发生旋转的情况下，将所述位图转化为图片字节流，并将所述图片字节流传输到所述第二终端以同步显示与所述位图对应的屏幕截图信息。
可选地，计算机程序被处理器执行时还实现如下步骤的程序代码：
S1，获取视窗管理对象的预设旋转函数的返回值，其中，所述返回值用于指示所述屏幕是否旋转以及旋转角度；
S2，按照所述返回值对所述位图执行与所述旋转状态对应的转置操作。
可选地，存储介质还被设置为存储用于执行以下步骤的计算机程序：
S1，在获取第一终端的屏幕截图信息之前，触发所述第二终端启动第一线程；
S2，基于所述第一线程建立所述第一终端与所述第二终端之间的套接字连接。
可选地，计算机程序被处理器执行时还实现如下步骤的程序代码：
S1，在获取第一终端的屏幕截图信息之前，触发所述第二终端启动第二线程，其中，所述第二线程用于在获取所述第一终端的屏幕截图信息失败的情况下，重启获取所述第一终端的屏幕截图信息的操作。
通过上述实施方式，在根据旋转状态确定屏幕发生旋转的情况下，对由屏幕截图信息转化的位图执行与旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将图片字节流传输到第二终端以同步显示与位图对应的屏幕截图信息，从而解决了相关技术在移动终端应用开发调试过程中，需要在PC端访问和操作远程终端屏幕，但是终端屏幕发生旋转，则同步显示的屏幕截图不完整的问题，达到了提高远程真机的同步显示效果。
计算机可读存储介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了可读程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读存储介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。
计算机可读存储介质中包含的程序代码可以用任何适当的介质传输，包括但不限于无线、有线、光缆、射频等等，或者上述的任意合适的组合。可选地，在本实施例中，上述存储介质可以包括但不限于：U盘、只读存储器(Read-Only Memory，简称为ROM)、随机存取存储器(Random Access Memory，简称为RAM)、移动硬盘、磁碟或者光盘等各种可以存储计算机程序的介质。
本发明本公开的实施例还提供了一种电子装置，包括存储器和处理器， 该存储器中存储有计算机程序，该处理器被设置为运行计算机程序以执行上述任一项方法实施例中的步骤，可以为如图1所示的移动终端。
可选地，上述电子装置还可以包括传输设备以及输入输出设备，其中，该传输设备和上述处理器连接，该输入输出设备和上述处理器连接。
可选地，在本实施例中，上述处理器可以被设置为通过计算机程序执行以下步骤：
S1，获取第一终端的屏幕截图信息，并将所述屏幕截图信息转化为对应的位图；
S2，获取所述第一终端的屏幕的旋转状态；
S3，在根据所述旋转状态确定所述屏幕发生旋转的情况下，对所述位图执行与所述旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将所述图片字节流传输到第二终端以同步显示与所述位图对应的屏幕截图信息；以及在根据所述旋转状态确定所述屏幕未发生旋转的情况下，将所述位图转化为图片字节流，并将所述图片字节流传输到所述第二终端以同步显示与所述位图对应的屏幕截图信息。
可选地，处理器还被设置为存储用于执行以下步骤的计算机程序：
S1，获取视窗管理对象的预设旋转函数的返回值，其中，所述返回值用于指示所述屏幕是否旋转以及旋转角度；
S2，按照所述返回值对所述位图执行与所述旋转状态对应的转置操作。
可选地，存储介质还被设置为存储用于执行以下步骤的计算机程序：
S1，在获取第一终端的屏幕截图信息之前，触发所述第二终端启动第一线程；
S2，基于所述第一线程建立所述第一终端与所述第二终端之间的套接字连接。
可选地，处理器还被设置为存储用于执行以下步骤的计算机程序：
S1，在获取第一终端的屏幕截图信息之前，触发所述第二终端启动第 二线程，其中，所述第二线程用于在获取所述第一终端的屏幕截图信息失败的情况下，重启获取所述第一终端的屏幕截图信息的操作。
可选地，本实施例中的具体示例可以参考上述实施例及可选实施方式中所描述的示例，本实施例在此不再赘述。
通过上述实施方式，在根据旋转状态确定屏幕发生旋转的情况下，对由屏幕截图信息转化的位图执行与旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将图片字节流传输到第二终端以同步显示与位图对应的屏幕截图信息，从而解决了相关技术在移动终端应用开发调试过程中，需要在PC端访问和操作远程终端屏幕，但是终端屏幕发生旋转，则同步显示的屏幕截图不完整的问题，达到了提高远程真机的同步显示效果。
上述处理器可以通过运行存储在存储器内的软件程序以及组件，从而执行各种功能应用以及数据处理，即实现上述的同步显示方法。
显然，所描述的实施例仅仅是本申请一部分的实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都应当属于本申请保护的范围。
需要说明的是，本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的本申请的实施例。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或组件的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或组件，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或组件。
显然，本领域的技术人员应该明白，上述的本公开的各组件或各步骤可以用通用的计算装置来实现，它们可以集中在单个的计算装置上，或者分布在多个计算装置所组成的网络上，可选地，它们可以用计算装置可执 行的程序代码来实现，从而，可以将它们存储在存储装置中由计算装置来执行，并且在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤，或者将它们分别制作成各个集成电路模块，或者将它们中的多个模块或步骤制作成单个集成电路模块来实现。这样，本公开不限制于任何特定的硬件和软件结合。
以上所述仅为本公开的优选实施例而已，并不用于限制本公开，对于本领域的技术人员来说，本公开可以有各种更改和变化。凡在本公开的原则之内，所作的任何修改、等同替换、改进等，均应包含在本公开的保护范围之内。
工业实用性
通过本公开，在根据旋转状态确定屏幕发生旋转的情况下，对由屏幕截图信息转化的位图执行与旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将图片字节流传输到第二终端以同步显示与位图对应的屏幕截图信息，从而解决了相关技术在移动终端应用开发调试过程中，需要在PC端访问和操作远程终端屏幕，但是终端屏幕发生旋转，则同步显示的屏幕截图不完整的问题，达到了提高远程真机的同步显示效果。

Claims (10)

  1. 一种同步显示方法，包括：
    获取第一终端的屏幕截图信息，并将所述屏幕截图信息转化为对应的位图；
    获取所述第一终端的屏幕的旋转状态；
    在根据所述旋转状态确定所述屏幕发生旋转的情况下，对所述位图执行与所述旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将所述图片字节流传输到第二终端以同步显示与所述位图对应的屏幕截图信息；以及在根据所述旋转状态确定所述屏幕未发生旋转的情况下，将所述位图转化为图片字节流，并将所述图片字节流传输到第二终端以同步显示与所述位图对应的屏幕截图信息。
  2. 根据权利要求1所述的方法，其中，在确定所述屏幕发生旋转的情况下，对所述位图进行与所述旋转状态对应的转置操作包括：
    获取视窗管理对象的预设旋转函数的返回值，其中，所述返回值用于指示所述屏幕是否旋转以及旋转角度；
    按照所述返回值对所述位图执行与所述旋转状态对应的转置操作。
  3. 根据权利要求1所述的方法，其中，在获取第一终端的屏幕截图信息之前，所述方法还包括：
    触发所述第二终端启动第一线程；
    基于所述第一线程建立所述第一终端与所述第二终端之间的套接字连接。
  4. 根据权利要求3所述的方法，其中，将进行转置操作后的位图传输到第二终端以同步显示与所述位图对应的屏幕截图信息包括：
    将进行转置操作后的位图转化成图片格式的字节流，并对所述字节流进行封装；
    通过所述套接字连接将封装后的字节流传输到所述第二终端以同步显示与所述字节流对应的屏幕截图信息。
  5. 根据权利要求1所述的方法，其中，在获取第一终端的屏幕截图信息之前，所述方法还包括：
    触发所述第二终端启动第二线程，其中，所述第二线程用于在获取所述第一终端的屏幕截图信息失败的情况下，重启获取所述第一终端的屏幕截图信息的操作。
  6. 一种同步显示装置，所述装置包括一个或多个处理器，以及一个或多个存储程序单元的存储器，其中，所述程序单元由所述处理器执行，所述程序单元包括：
    第一获取组件，设置为获取第一终端的屏幕截图信息，并将所述屏幕截图信息转化为对应的位图；
    第二获取组件，设置为获取所述第一终端的屏幕的旋转状态；
    显示组件，设置为在根据所述旋转状态确定所述屏幕发生旋转的情况下，对所述位图执行与所述旋转状态对应的转置操作，将转置操作后的位图转化为图片字节流，并将所述图片字节流传输到第二终端以同步显示与所述位图对应的屏幕截图信息；以及在根据所述旋转状态确定所述屏幕未发生旋转的情况下，将所述位图转化为图片字节流，并将所述图片字节流传输到第二终端以同步显示与所述位图对应的屏幕截图信息。
  7. 根据权利要求6所述的装置，其中，所述显示组件包括：
    获取元件，设置为获取视窗管理对象的预设旋转函数的返回值，其中，所述返回值用于指示所述屏幕是否旋转以及旋转角度；
    转置元件，设置为按照所述返回值对所述位图执行与所述旋转状态对应的转置操作。
  8. 根据权利要求6所述的装置，其中，所述装置还包括：
    第一触发组件，设置为在获取第一终端的屏幕截图信息之前，触发所述第二终端启动第一线程；
    建立组件，设置为基于所述第一线程建立所述第一终端与所述第二终端之间的套接字连接。
  9. 一种存储介质，所述存储介质中存储有计算机程序，其中，所述计算机程序被设置为运行时控制所述存储介质所在设备执行所述权利要求1至5任一项中所述的方法。
  10. 一种电子装置，包括存储器和处理器，其中，所述存储器中存储有计算机程序，所述处理器被设置为运行所述计算机程序以执行所述权利要求1至5任一项中所述的方法。
PCT/CN2019/089770 2019-01-17 2019-06-03 同步显示方法及装置、存储介质和电子装置 WO2020147246A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/042,938 US11340859B2 (en) 2019-01-17 2019-06-03 Synchronous display method, storage medium and electronic device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910045548.0A CN109766163B (zh) 2019-01-17 2019-01-17 同步显示方法及装置、存储介质和电子装置
CN201910045548.0 2019-01-17

Publications (1)

Publication Number Publication Date
WO2020147246A1 true WO2020147246A1 (zh) 2020-07-23

Family

ID=66452508

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/089770 WO2020147246A1 (zh) 2019-01-17 2019-06-03 同步显示方法及装置、存储介质和电子装置

Country Status (3)

Country Link
US (1) US11340859B2 (zh)
CN (1) CN109766163B (zh)
WO (1) WO2020147246A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113189920A (zh) * 2021-05-28 2021-07-30 重庆长安汽车股份有限公司 一种基于Android的车载娱乐信息系统远程控制方法

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109766163B (zh) * 2019-01-17 2020-02-07 网易（杭州）网络有限公司 同步显示方法及装置、存储介质和电子装置
CN112364616B (zh) * 2019-07-26 2024-04-30 珠海金山办公软件有限公司 一种电子表格的处理方法、装置、电子设备及存储介质
CN110716778B (zh) * 2019-09-10 2023-09-26 创新先进技术有限公司 应用兼容性测试方法、装置及系统
CN110806846A (zh) * 2019-10-11 2020-02-18 北京字节跳动网络技术有限公司 屏幕共享方法、屏幕共享装置、移动终端、存储介质
CN111427524A (zh) * 2020-02-27 2020-07-17 深圳壹账通智能科技有限公司 远程控制的方法、装置、计算机设备及存储介质
CN113360335A (zh) * 2020-03-06 2021-09-07 北京沃东天骏信息技术有限公司 监控终端的方法、装置、设备和计算机可读介质
CN112468871B (zh) * 2020-11-24 2022-10-14 青岛创新奇智科技集团股份有限公司 监控视频播放方法、装置及电子设备
CN113014996B (zh) * 2021-02-18 2022-07-22 上海哔哩哔哩科技有限公司 视频生成方法及装置
TWI842456B (zh) * 2023-03-27 2024-05-11 大陸商星宸科技股份有限公司 電子裝置及其設備資源管理方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120185066A1 (en) * 2011-01-18 2012-07-19 Mark Kern Systems and methods for generating enhanced screenshots
CN105988688A (zh) * 2016-01-22 2016-10-05 腾讯科技（深圳）有限公司 一种截图的方法及装置
CN108347452A (zh) * 2017-01-23 2018-07-31 腾讯科技（深圳）有限公司 远程获取屏幕截图的方法及装置
CN109766163A (zh) * 2019-01-17 2019-05-17 网易（杭州）网络有限公司 同步显示方法及装置、存储介质和电子装置

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2001067245A2 (en) * 2000-03-08 2001-09-13 Marbles, Inc. Remote operation of real-time graphical applications
US8190359B2 (en) * 2007-08-31 2012-05-29 Proxpro, Inc. Situation-aware personal information management for a mobile device
US10445051B1 (en) * 2014-03-27 2019-10-15 Amazon Technologies, Inc. Recording and replay of support sessions for computing devices
CN104615300B (zh) * 2014-12-29 2018-12-11 合肥杰发科技有限公司 图像接收装置及其对电子装置的屏幕放置状态的判断方法
JP6122477B2 (ja) * 2015-10-15 2017-04-26 株式会社オプティム 画面共有システム及び画面共有方法
KR20200136753A (ko) * 2019-05-28 2020-12-08 삼성전자주식회사 외부 전자 장치를 통해 화면 공유 서비스를 제공하기 위한 전자 장치, 방법, 및 컴퓨터 판독가능 매체
US20200380935A1 (en) * 2019-05-31 2020-12-03 Apple Inc. Automatic Display Orientation Configuration

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120185066A1 (en) * 2011-01-18 2012-07-19 Mark Kern Systems and methods for generating enhanced screenshots
CN105988688A (zh) * 2016-01-22 2016-10-05 腾讯科技（深圳）有限公司 一种截图的方法及装置
CN108347452A (zh) * 2017-01-23 2018-07-31 腾讯科技（深圳）有限公司 远程获取屏幕截图的方法及装置
CN109766163A (zh) * 2019-01-17 2019-05-17 网易（杭州）网络有限公司 同步显示方法及装置、存储介质和电子装置

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113189920A (zh) * 2021-05-28 2021-07-30 重庆长安汽车股份有限公司 一种基于Android的车载娱乐信息系统远程控制方法

Also Published As

Publication number Publication date
US11340859B2 (en) 2022-05-24
CN109766163A (zh) 2019-05-17
CN109766163B (zh) 2020-02-07
US20210334061A1 (en) 2021-10-28

Similar Documents

Publication Publication Date Title
WO2020147246A1 (zh) 同步显示方法及装置、存储介质和电子装置
CN112799891B (zh) iOS设备测试方法、装置、系统、存储介质及计算机设备
CN108874677B (zh) 一种安卓终端及其测试方法和系统
CN104539682A (zh) 移动网页的调试方法、装置、移动终端、服务器以及系统
CN104253842B (zh) 同步终端镜像的方法、装置、终端及服务器
CN110933075B (zh) 服务调用方法、装置、电子设备及存储介质
CN113893524B (zh) 云应用处理系统、方法、装置及设备
WO2024037110A9 (zh) 数据处理方法、装置、设备以及介质
CN108347452A (zh) 远程获取屏幕截图的方法及装置
CN111444095A (zh) 移动设备的测试平台系统和测试方法
CN109788251B (zh) 视频处理方法、装置及存储介质
CN104570967B (zh) 基于Android系统的远程控制方法及系统
CN114048087B (zh) 设备的数据中转性能的测试方法及装置
CN110753363A (zh) 一种智能设备无线调试系统和方法
CN109634856A (zh) 基于云服务的iOS智能终端远程真机调试系统及方法
CN108737337B (zh) 设备处理方法、装置、服务器及系统
CN113391931B (zh) 基于蓝牙的远程控制方法、装置、计算机设备和存储介质
CN115567431A (zh) 一种网络稳定性测试方法、装置、被测设备及存储介质
CN113053374B (zh) 一种大屏幕控制系统
CN112671814B (zh) 一种跨平台的设备开发方法、装置及系统
TW202321913A (zh) 虛擬行動架構的遠端渲染系統、方法及裝置
CN114095758A (zh) 云图像截取方法、相关装置及计算机程序产品
CN111698333A (zh) 远程控制方法、服务器和系统
Wang et al. Design and implementation of remote control system between Android platform
CN116954809B (zh) 基于安卓设备的远程控制方法、装置、系统及电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19909827

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19909827

Country of ref document: EP

Kind code of ref document: A1