WO2021233218A1 - 投屏方法、投屏源端、投屏目的端、投屏系统及存储介质 - Google Patents

投屏方法、投屏源端、投屏目的端、投屏系统及存储介质 Download PDF

Info

Publication number
WO2021233218A1
WO2021233218A1 PCT/CN2021/093768 CN2021093768W WO2021233218A1 WO 2021233218 A1 WO2021233218 A1 WO 2021233218A1 CN 2021093768 W CN2021093768 W CN 2021093768W WO 2021233218 A1 WO2021233218 A1 WO 2021233218A1
Authority
WO
WIPO (PCT)
Prior art keywords
interface
screen projection
call
image
user
Prior art date
Application number
PCT/CN2021/093768
Other languages
English (en)
French (fr)
Inventor
张创
高蕾
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021233218A1 publication Critical patent/WO2021233218A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems

Definitions

  • This application relates to the field of terminal technology, and in particular to a screen projection method, a projection source terminal, a projection destination terminal, a projection system, and a computer-readable storage medium.
  • Multi-screen interactive technology is a technology for sharing screen display content between different terminal devices.
  • the multi-screen sharing of the video call (for example, video chat, multi-person video conference, etc.) interface is an important application of the multi-screen interactive technology.
  • Some embodiments of the application provide a projection method, a projection source, a projection destination, and a projection system.
  • the following introduces the application from multiple aspects.
  • the implementations and beneficial effects of the following multiple aspects can be referred to each other. .
  • the embodiments of the present application provide a screen projection method, which is applied to a screen projection source terminal with a video call function.
  • the screen projection method includes: in a video call state, receiving the user's first call sent by the projection destination terminal Image; generate a video call interface for display at the source end of the screen, the video call interface includes the user’s first call image and interface elements; send the image data of the interface elements to the destination end of the screen so that the destination end of the screen will display the interface elements and
  • the user’s second call image is synthesized to generate an interface for display at the projection destination; wherein the user’s second call image is an image collected by the camera of the projection destination when the projection destination receives the image data of the interface elements.
  • the screen projection destination combines the interface elements of the video call interface of the screen projection source with the real-time user call image collected by the local camera to generate a video call interface for display at the screen projection destination.
  • the collection time of the user's call image and the display time are only separated by the time the screen projection destination processes the image data, which can significantly reduce the user's perceived delay.
  • the interface elements of the video call interface include at least one of the following: controls for controlling the call status and/or interface display status, call status information, user identification, and calls of other users participating in the video call image.
  • the image data of the interface element includes: image information of each element in the interface element, and position information of each element on the video call interface. According to the embodiments of the present application, the amount of image data of interface elements can be reduced.
  • the start condition of the projection method is: the communication delay between the projection source and the projection destination exceeds a set threshold; and/or, the delay between the projection source and the projection destination The amount of data buffer in the communication channel exceeds the set threshold.
  • the embodiments of the present application provide a screen projection method, which is applied to the screen projection destination, and the screen projection method includes: when the projection source terminal is in a video call state, sending the user's first call image to the projection source terminal , So that the projection source terminal generates a video call interface for display on the projection source terminal.
  • the video call interface includes the user’s first call image and interface elements; receives the image data of the interface elements sent by the projection source terminal; combines the interface elements and
  • the user’s second call image is synthesized to generate an interface for display at the projection destination; wherein the user’s second call image is an image collected by the camera of the projection destination when the projection destination receives the image data of the interface elements.
  • the screen projection destination combines the interface elements of the video call interface of the screen projection source with the real-time user call image collected by the local camera to generate a video call interface for display at the screen projection destination.
  • the collection time of the user's call image and the display time are only separated by the time the screen projection destination processes the image data, which can significantly reduce the user's perceived delay.
  • the interface elements of the video call interface include at least one of the following elements: controls for controlling the call status and/or interface display status, call status information, user identification, and information of other users participating in the video call. Call image.
  • the image data of the interface element includes: image information of each element in the interface element, and position information of each element on the video call interface. According to the embodiments of the present application, the data amount of image data of interface elements can be reduced.
  • the start condition of the projection method is: the communication delay between the projection source and the projection destination exceeds a set threshold; and/or, the delay between the projection source and the projection destination The amount of data buffer in the communication channel exceeds the set threshold.
  • the embodiments of the present application provide a screen projection source terminal, which has a video call function, and the projection screen source terminal includes a memory for storing data executed by one or more processors of the projection source terminal. Instruction; the processor, when the processor executes the instructions in the memory, it can cause the screen projection source to execute the screen projection method provided by any one of the implementations of the first aspect of this application.
  • the processor when the processor executes the instructions in the memory, it can cause the screen projection source to execute the screen projection method provided by any one of the implementations of the first aspect of this application.
  • the embodiments of the present application provide a screen projection destination, the projection destination has a video call function, and the projection destination includes a memory for storing data executed by one or more processors of the projection destination.
  • the beneficial effects that can be achieved in the fourth aspect reference may be made to the beneficial effects of the method provided by any embodiment of the second aspect, which will not be repeated here.
  • the embodiments of this application provide a screen projection system.
  • the screen projection system includes a screen projection source terminal and a projection destination terminal.
  • the projection screen source terminal has a video call function, and the projection screen source terminal is used to execute this application.
  • the projection destination is used to execute the projection method provided by any embodiment of the second aspect of the present application.
  • the embodiments of the present application provide a computer-readable storage medium.
  • the computer-readable storage medium stores instructions.
  • the computer can execute any of the Screen projection method, or make the computer execute the projection method provided by any one of the implementation manners of the second aspect of this application.
  • the beneficial effects that can be achieved in the sixth aspect reference may be made to the beneficial effects of the method provided in any implementation manner of the first aspect or any implementation manner of the second aspect, which will not be repeated here.
  • FIG. 1 is an exemplary application scenario of the screen projection method provided by the embodiment of this application.
  • FIG. 2 is a schematic diagram of an image data transmission process provided by an embodiment of this application.
  • FIG. 3 is a schematic diagram of an image data transmission process in the prior art
  • FIG. 4 is a schematic diagram of the structure of an electronic device provided by an embodiment of the application.
  • FIG. 5a is a software architecture diagram of the source end of the projection screen provided by the embodiment of this application.
  • FIG. 5b is a software architecture diagram of the screen projection destination provided by the embodiment of this application.
  • Fig. 6a is a schematic diagram of a multi-screen interactive scene provided by an embodiment of this application (the source end of the projection screen);
  • FIG. 6b is a schematic diagram of a multi-screen interactive scene provided by an implementation manner of this application (the screen projection destination);
  • FIG. 7 is a schematic flowchart of a screen projection method provided by an embodiment of this application.
  • FIG. 8 is a schematic diagram of a mobile phone side interface provided by an embodiment of this application.
  • FIG. 9a is a schematic diagram of interface element image data of a mobile phone side interface provided by an embodiment of this application.
  • FIG. 9b is another schematic diagram of the interface element image data of the mobile phone side interface provided by the embodiment of this application.
  • FIG. 9c is a schematic diagram of a tablet side interface according to an embodiment of the application.
  • FIG. 10 is an exemplary structure diagram of a screen projection system provided by an embodiment of this application.
  • FIG. 11 shows a block diagram of an electronic device provided by an embodiment of the present application.
  • FIG. 12 shows a schematic structural diagram of a system on chip (SoC, System on Chip) provided by an embodiment of the present application.
  • SoC System on Chip
  • Figure 1 shows a video call between user A and user B.
  • User A owns device A1 and device A2, where a video call application (for example, WeChat, Skype, etc.) is installed on device A1, so that device A1 can generate a video call interface through the video call application during the video call.
  • a video call application for example, WeChat, Skype, etc.
  • the device A1 can transmit the interface image of the video call interface to the device A2 (for example, transmission through screen projection technology, or transmission based on a distributed system).
  • the equipment A2 can display the video call interface generated by device A1.
  • the video call interface includes the call image of user A, the call image of user B, interface controls, call status information, and so on.
  • device A2 is the device currently operated by user A (for example, user A is using an application (for example, an image processing application) on device A2, or user A needs to pass through when driving a vehicle Device A2 (for example, a car machine) makes a video call, etc.).
  • the call image of user A on the video call interface is collected by the camera of device A2.
  • the device A1 After the camera of device A2 collects the call image of user A, it will The call image is transmitted to the device A1, and the device A1 generates a video call interface based on the call image transmitted from the device A2.
  • the device A1 transmits the generated interface image of the video call interface to the device A2, so that the user A who is operating the device A2 can see the video call interface.
  • the embodiments of the present application are used to provide a screen projection method applied to the scenario shown in FIG. 1 to reduce user perception delay.
  • the device A2 does not directly display the video call interface on the device A1, but synthesizes the interface element image in the video call interface of A1 with the user call image collected by the local camera in real time to generate It is used for the video call interface displayed by device A2.
  • the user call image is a real-time image of the user A, which is basically consistent with the current state of the user A, which can significantly reduce the user perception delay and improve the user experience.
  • the device A1 is the source end of the screen projection
  • the device A2 is the destination end of the screen projection.
  • the mobile phone is used as an example of the screen projection source and the tablet is used as an example of the screen projection destination. Therefore, the video call interface generated by the mobile phone is called the “mobile phone side interface”, and the video call interface generated by the tablet is called It is the "flat-side interface”.
  • the screen projection source can be other devices with video call functions, such as tablets, laptops, etc.; the screen projection destination can be other devices with display functions.
  • the projection source and the projection destination can be the same type of equipment (for example, the projection source and the projection destination are both mobile phones), or they can be different types of equipment (for example, the projection source The end is a mobile phone, and the destination of the screen is a car machine).
  • the number of users participating in a video call is not limited, and it can be a one-to-one video call, such as the video call scene between user A and user B shown in Figure 1; it can also be many-to-many.
  • Video calls for example, multi-party video conference scenes, online teaching scenes, etc.
  • Fig. 2 shows the image data transmission process during the video call in the embodiment of the present application.
  • device A2 collects and sends the call image P1 of user A to device A1;
  • device A1 receives the call image P1 of user A sent by device A2, and the call image of user B sent by device B;
  • device A1 sends to device A2 the interface element image of the video call interface (ie, the mobile phone interface) generated by device A1 (which does not include user A's call image P1);
  • device A2 receives device A1
  • the interface element image is sent, and the call image P2 of user A is collected in real time through the camera of device A2, and then the interface element image and the call image P2 of user A are synthesized to generate a video call interface (ie, tablet Side interface);
  • the device A2 displays the tablet side interface, and the tablet side interface includes the call image P2 of user A.
  • Figure 3 shows an existing technical solution in the prior art.
  • device A1 generates a video call interface based on user A’s call image P1 and user B’s call image (ie, Mobile phone interface), and send the interface image of the mobile phone interface to device A2;
  • device A2 receives the interface image of the mobile phone interface;
  • device A2 displays the mobile phone interface, which includes device A2 at time T1 The collected call image P1 of user A.
  • the main cause of the user's perception delay is the transmission delay of image data in the communication network and the processing delay of the terminal device processing the image data, among which the former is the main cause of the user's perception delay.
  • the user perception delay may exceed 300ms.
  • the user perception delay may exceed 500ms, which will reduce the user experience.
  • device A2 synthesizes the interface elements of the video call interface of device A1 with the user's call image collected in real time by the local camera to generate the video call interface displayed by device A2, that is, device A2 On the displayed video call interface, user A’s call image P2 is the image captured by the camera of device A2 at time T4', because the call image P2’s collection time (ie, T4') and the display time (ie, T5') are different Only the time for device A2 to process the image data is separated from each other, which can significantly reduce the user's perceived delay.
  • FIG. 4 shows a schematic diagram of the structure of the electronic device 100.
  • the electronic device 100 can be implemented as a mobile phone or a tablet.
  • the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) connector 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2.
  • Mobile communication module 150 wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (subscriber identification module, SIM) card interface 195, etc.
  • SIM Subscriber identification module
  • the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light Sensor 180L, bone conduction sensor 180M, etc.
  • the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100.
  • the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or arrange different components.
  • the illustrated components can be implemented in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units.
  • the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU), etc.
  • AP application processor
  • modem processor modem processor
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • NPU neural-network processing unit
  • the different processing units may be independent devices or integrated in one or more processors.
  • the processor can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.
  • a memory may also be provided in the processor 110 to store instructions and data.
  • the memory in the processor 110 is a cache memory.
  • the memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.
  • the processor 110 may include one or more interfaces.
  • Interfaces can include integrated circuit (I2C) interfaces, universal asynchronous receiver/transmitter (UART) interfaces, mobile industry processor interfaces (MIPI), general input and output (general input and output) interfaces. -purpose input/output, GPIO) interface, subscriber identity module (SIM) interface, and/or universal serial bus (universal serial bus, USB) interface, etc.
  • I2C integrated circuit
  • UART universal asynchronous receiver/transmitter
  • MIPI mobile industry processor interfaces
  • GPIO GPIO
  • SIM subscriber identity module
  • USB universal serial bus
  • the I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL).
  • the processor 110 may include multiple sets of I2C buses.
  • the processor 110 may be coupled to the camera 193 through an I2C bus interface.
  • the UART interface is a universal serial data bus used for asynchronous communication.
  • the bus can be a two-way communication bus. It will convert the data to be transmitted between serial communication and parallel communication.
  • the UART interface is generally used to connect the processor 110 and the wireless communication module 160.
  • the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to realize the Bluetooth function.
  • the MIPI interface can be used to connect the processor 110 with the display screen 194, the camera 193 and other peripheral devices.
  • the MIPI interface includes a camera serial interface (camera serial interface, CSI), a display serial interface (display serial interface, DSI), and so on.
  • the processor 110 and the camera 193 communicate through a CSI interface to implement the shooting function of the electronic device 100.
  • the processor 110 and the display screen 194 communicate through a DSI interface to realize the display function of the electronic device 100.
  • the GPIO interface can be configured through software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, and so on.
  • the GPIO interface can also be configured as an I2C interface, UART interface, MIPI interface, etc.
  • the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely a schematic description, and does not constitute a structural limitation of the electronic device 100.
  • the electronic device 100 may also adopt different interface connection modes in the foregoing embodiments, or a combination of multiple interface connection modes.
  • the wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.
  • the antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.
  • Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
  • the antenna can be used in combination with a tuning switch.
  • the mobile communication module 150 can provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the electronic device 100.
  • the mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like.
  • the mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform processing such as filtering, amplifying and transmitting the received electromagnetic waves to the modem processor for demodulation.
  • the mobile communication module 150 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves for radiation via the antenna 1.
  • at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110.
  • at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.
  • the modem processor may include a modulator and a demodulator.
  • the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal.
  • the demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194.
  • the modem processor may be an independent device.
  • the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.
  • the wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), and global navigation satellites.
  • WLAN wireless local area networks
  • BT wireless fidelity
  • GNSS global navigation satellite system
  • FM frequency modulation
  • NFC near field communication technology
  • infrared technology infrared, IR
  • the wireless communication module 160 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110.
  • the wireless communication module 160 may also receive a signal to be sent from the processor 110, perform frequency modulation, amplify, and convert it into electromagnetic waves to radiate through the antenna 2.
  • the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc.
  • the GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite-based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite-based augmentation systems
  • the electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc.
  • the audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal.
  • the audio module 170 can also be used to encode and decode audio signals.
  • the audio module 170 may be provided in the processor 110, or part of the functional modules of the audio module 170 may be provided in the processor 110.
  • the electronic device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like.
  • the GPU is an image processing microprocessor, which is connected to the display screen 194 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations and is used for graphics rendering.
  • the processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
  • the display screen 194 is used to display images, videos, and the like.
  • the display screen 194 includes a display panel.
  • the display panel can use liquid crystal display (LCD), organic light-emitting diode (OLED), active matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • active-matrix organic light-emitting diode active-matrix organic light-emitting diode
  • AMOLED flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.
  • the electronic device 100 may include one or N display screens 194, and N is a positive integer greater than one.
  • the electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.
  • the ISP is used to process the data fed back from the camera 193. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing and is converted into an image visible to the naked eye.
  • ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be provided in the camera 193.
  • the camera 193 is used to capture still images or videos.
  • the object generates an optical image through the lens and is projected to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the optical signal into an electrical signal, and then transfers the electrical signal to the ISP to convert it into a digital image signal.
  • ISP outputs digital image signals to DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other formats of image signals.
  • the electronic device 100 may include one or N cameras 193, and N is a positive integer greater than one.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects the frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.
  • MPEG moving picture experts group
  • MPEG2 MPEG2, MPEG3, MPEG4, and so on.
  • NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • applications such as intelligent cognition of the electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, and so on.
  • the external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, save music, video and other files in an external memory card.
  • the internal memory 121 may be used to store computer executable program code, where the executable program code includes instructions.
  • the internal memory 121 may include a program storage area and a data storage area.
  • the storage program area can store an operating system, an application program (such as a sound playback function, an image playback function, etc.) required by at least one function, and the like.
  • the data storage area can store data (such as audio data, phone book, etc.) created during the use of the electronic device 100.
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.
  • UFS universal flash storage
  • the internal memory 121 and/or the memory provided in the processor stores instructions.
  • the electronic device 100 is implemented as a mobile phone
  • the instructions stored in the internal memory 121 and/or the memory provided in the processor are executed by the processor 110
  • the mobile phone can be executed by the mobile phone in the projection method provided in the embodiments of the present application.
  • the electronic device 100 is implemented as a tablet
  • the instructions stored in the internal memory 121 and/or the memory provided in the processor are executed by the processor 110
  • the tablet can execute the projection method provided in the embodiments of this application Steps performed by the tablet.
  • the electronic device 100 may include more or less components than those shown in FIG. 4, which is not limited in the embodiment of the present application.
  • the illustrated electronic device 100 is only an example, and the device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations.
  • the various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and/or application specific integrated circuits.
  • the mobile phone software system can adopt a layered architecture, event-driven architecture, micro-core architecture, micro-service architecture, or cloud architecture.
  • the embodiment of the present invention takes an Android system with a layered architecture as an example to exemplify the software structure of the mobile phone system architecture.
  • Figure 5a shows the system architecture of the mobile phone.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Communication between layers through software interface.
  • the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.
  • the application layer can include a series of application packages.
  • the application package on the mobile phone can include video call applications, mobile screen projection applications, cameras, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.
  • the video call application is used to generate a mobile-side interface during a video call. Specifically, during a video call, the video call application obtains the call image of user A and the call image of user B, and generates call controls (for example, controls that control the call state, controls that control the display state of the interface), The user identification, etc., afterwards, the video call application synthesizes the call image of user A, the call image of user B, call controls, user identification, etc., to form a mobile phone side interface.
  • other elements on the interface of the mobile phone side except for the call image of user A are referred to as interface elements.
  • the video call application also transmits the image data of the interface elements of the mobile phone side interface to the mobile screen projection application.
  • the mobile phone projection application processes the image data of the interface element, for example, encoding, packaging, etc., and then the mobile phone sends the processed image data to the tablet.
  • the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, and a notification manager.
  • the window manager is used to manage window programs.
  • the window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take a screenshot, etc.
  • the content provider is used to store and retrieve data and make these data accessible to applications.
  • the data may include videos, images, audios, phone calls made and received, browsing history and bookmarks, phone book, etc.
  • the view system includes visual controls, such as controls that display text, controls that display pictures, and so on.
  • the view system can be used to build applications.
  • the display interface can be composed of one or more views.
  • a display interface that includes a short message notification icon may include a view that displays text and a view that displays pictures.
  • the phone manager is used to provide the communication function of the mobile phone. For example, the management of the call status (including connecting, hanging up, etc.).
  • the resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.
  • the notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and it can automatically disappear after a short stay without user interaction.
  • the notification manager is used to notify download completion, message reminders, and so on.
  • the notification manager can also be a notification that appears in the status bar at the top of the system in the form of a chart or a scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, text messages are prompted in the status bar, prompt sounds, electronic devices vibrate, and indicator lights flash.
  • Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.
  • the core library consists of two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.
  • the application layer and application framework layer run in a virtual machine.
  • the virtual machine executes the java files of the application layer and the application framework layer as binary files.
  • the virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
  • the system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), three-dimensional graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.
  • the surface manager is used to manage the display subsystem and provides a combination of 2D and 3D layers for multiple applications.
  • the media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files.
  • the media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
  • the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing.
  • the 2D graphics engine is a drawing engine for 2D drawing.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer contains at least display driver, camera driver, audio driver, sensor driver, etc.
  • Figure 5b shows the system architecture of the tablet.
  • the system architecture of the tablet is basically the same as the system architecture of the mobile phone. The following only introduces the differences between the tablet system architecture and the mobile phone system architecture, and the same parts of the two will not be repeated.
  • the application layer of the tablet includes tablet projection applications and mobile phone clone applications.
  • the flat screen projection application decodes the received image data to restore the interface element image of the mobile phone side interface.
  • the tablet-side projection screen can also obtain the user's call image P2 collected by the tablet camera, and synthesize the interface element image of the tablet-side interface with the user's call image P2 to generate a superimposed image.
  • the mobile phone clone application can obtain the superimposed image generated by the tablet projection application, and process the superimposed image, for example, adjust the resolution of the superimposed image, and add the identification information of the projection source (ie, mobile phone) to the superimposed image, etc. , To generate the flat side interface.
  • the projection source ie, mobile phone
  • user A’s mobile phone and tablet are in a multi-screen interactive state, that is, the mobile phone shares its screen display content on the tablet through the multi-screen interactive technology, and the tablet displays the mobile phone screen through its installed mobile phone avatar application Content.
  • the mobile phone avatar application on the tablet displays a list of applications on the mobile phone.
  • one party of the video call initiates a video call request to the other party.
  • the tablet sends user A’s operation information (for example, operation location, operation type, etc.) to the mobile phone, and the mobile phone determines the user’s instructions based on the operation information sent by the tablet To open the video call application.
  • the mobile phone starts the video call application and displays a contact interface that can establish a video call connection.
  • the tablet displays the contact interface of the video call application simultaneously.
  • the mobile phone can send a video call request to user B's device (ie, device B).
  • user B sends an instruction to accept the video call request to his video call device, a video call connection is established between user A's mobile phone and user B's device.
  • this embodiment is used to provide a screen projection method in the video call state, including the following steps:
  • the mobile phone obtains the call image P1 of user A (as the user's first call image) and the call image of user B, where the call image of user A is the image collected by the tablet camera, and the call image of user B An image collected by user B’s device.
  • the mobile phone After the mobile phone establishes a video call connection with user B's device, the mobile phone sends a call image acquisition request to the tablet (for example, the mobile phone sends a video call connection establishment signal to the tablet).
  • the tablet In response to the call image acquisition request sent by the mobile phone, the tablet activates the camera and shoots the call image of user A at a certain frame rate (for example, 40 fps). In the current step, the tablet sends the call image P1 of user A captured by the camera to the mobile phone.
  • the device of user B collects the call image of user B, and sends the call image of user B to the mobile phone.
  • the call image of the user is an image captured by the camera of the device, and is not necessarily an image of the user himself.
  • the camera of the tablet faces the position of the demo board.
  • the call image of user A is the image of the demo board.
  • the tablet can also collect user A’s voice signal and send user A’s voice signal to the mobile phone.
  • the mobile phone sends user A’s voice signal to user B’s device through a video call application, so that user B can hear user A’s device. the sound of.
  • the device of user B collects the voice signal of user B and sends the voice signal of user B to the mobile phone, and the mobile phone sends the voice signal of user B back to the tablet so that user A can hear user B's voice.
  • the mobile phone generates a mobile phone-side interface (as a video call interface for display at the source end of the projection screen).
  • the mobile phone generates the mobile phone-side interface through a video call application.
  • the mobile phone video application determines the call controls, call information, etc. on the video call interface displayed on the mobile phone, and synthesizes the call controls, call information, etc. with the call image P1 of user A and the call image of user B to generate Mobile phone side interface.
  • the elements on the mobile phone side interface other than the call image of the user A are referred to as the "interface elements" of the mobile phone side interface.
  • the interface elements of the mobile phone side interface include one or more of the following: controls for controlling the call state, for example, the hang up button 10; the voice call/video call switch button 20, etc.; Controls used to control the display status of the interface, such as front camera/rear camera switch button 30, call image switch button (used to switch the display position of user A's call image and user B's call image), etc.; call status information For example, the call duration is 40, etc.; the call image 50 of user B (as the call image of other users participating in the video call), etc.
  • the interface elements may include more or fewer elements than this embodiment.
  • the interface elements may also include user identifications (for example, user A and/or user B). Avatars and nicknames, etc.), interface borders, etc.
  • the call image P1 of user A is displayed in the upper right corner of the mobile phone side interface, and the call image of user B is distributed on the entire mobile phone side interface.
  • the call image of the user and the call image of the user B may have different distribution methods.
  • the call image of the user A and the call image of the user B switch positions according to the setting in FIG. 8, or, The call image of user A and the call image of user B are displayed side by side on the side interface of the mobile phone.
  • the mobile phone sends the image data of the interface elements to the tablet.
  • the video call application of the mobile phone determines the interface elements of the mobile phone side interface, and then transmits the image data of the interface elements to the mobile screen projection application.
  • the mobile screen projection application processes the image data of the interface elements, the processed image data Send the image data to the tablet.
  • the image data of the interface elements includes the image information of each interface element and the location information of each interface element on the mobile phone side interface.
  • the interface elements on the video call interface are numbered. Specifically, the interface elements are numbered as element 1, element 2, ..., element 6, respectively.
  • element 2 that is, the switch control used to control the on-off state of a video call
  • the image information of element 2 is the pixel value of each pixel forming element 2
  • the position information of element 2 is element 2 on the mobile phone interface. coordinate of.
  • the position information of the element 2 is the coordinates (35mm, 10mm) of its center point.
  • the image data of the interface element includes the image information and position information of each interface element. In this way, the amount of data of the image data of the interface element is relatively small.
  • the present application is not limited to this.
  • the image data of the interface element is the pixel value of each pixel in the virtual frame area.
  • the mobile phone video call application After the mobile phone video call application determines the image data of the interface elements, it sends the image data to the mobile screen projection application.
  • the mobile screen projection application processes the image data, for example, encodes the image data and follows the projection protocol (for example, the miracast protocol, The airplay mirroring protocol, etc.) performs encapsulation, etc., and then sends the processed interface element image data to the tablet.
  • the projection protocol for example, the miracast protocol, The airplay mirroring protocol, etc.
  • step S40 The tablet generates a superimposed image of the call image P2 of the user A (as the user's second call image) and the interface element image.
  • step S40 specifically includes:
  • the flat screen projection application restores the interface element image.
  • the tablet receives the image data of the interface elements sent by the mobile phone, it decapsulates and decodes the image data through the tablet projection application, restores the image data of each interface element, and restores the interface element image according to the image data of the interface element (That is, the image of the other part of the interface on the mobile phone side except for the user's call image P1).
  • the tablet projection application obtains the call image P2 of user A. After the tablet receives the image data of the interface elements sent by the mobile phone, the tablet camera collects the call image P2 of user A in real time, and the tablet projection application obtains the call image P2 of user A through the tablet's underlying system.
  • the tablet projection application superimposes the call image P2 of the user A with the interface element image to generate a superimposed image.
  • the tablet determines the size and position of user A's call image P2 on the superimposed image according to the size and position of user A's call image P1 on the mobile phone side interface (this information may be sent by the mobile phone to the tablet), and Make the superimposed image as close as possible to the phone side interface.
  • this application is not limited to this.
  • the size of user A's call image P2 and its position on the superimposed image are set by the user.
  • the tablet generates a tablet-side interface (as an interface displayed at the destination of the projection screen) according to the superimposed image.
  • the flat screen projection application transmits the generated superimposed image to the mobile phone clone application on the tablet, and the mobile phone clone application processes the superimposed image, for example, adjusts the resolution of the superimposed image to make the area of the tablet side interface Matching with the area of the flat panel display; and/or, adding identification information of the projection source end on the superimposed image (for example, adding the text "projected from Mate40" on the superimposed image), etc., to generate a flat-panel interface.
  • the mobile phone avatar application transmits the interface image of the tablet side interface to the tablet bottom system, so that the tablet bottom system controls the tablet display to display the tablet side interface (as shown in FIG. 9c).
  • the applications in this embodiment can be combined, split, or recombined.
  • the video call application on the mobile phone and the mobile screen projection application can be merged into one application; or, part of the flat screen projection application function (for example, the image overlay function in step S43) can be merged into the mobile phone avatar application on the tablet .
  • steps in this embodiment can be omitted, or steps can be added on the basis of this embodiment.
  • the step of processing the image data of the interface element by the mobile phone screen projection module in step S30 is omitted, and the mobile phone directly sends the image data of the interface element determined by the video call application to the tablet; or, the mobile phone avatar application pair in step S50 is omitted.
  • the tablet directly displays the superimposed image generated by the tablet projection application as the tablet side interface.
  • the screen projection method provided in this application can be set startup conditions for the screen projection method provided in this application.
  • the method in the prior art for example, the method shown in FIG. 3
  • the screen projection method provided in this application can be activated to reduce the user's perceived delay.
  • the communication delay can be determined according to the image transmission time between the tablet and the mobile phone.
  • the call image P1 of user A sent by the tablet to the mobile phone has a timestamp.
  • the mobile phone can obtain the transmission time T1 of the call image P1 by reading the timestamp.
  • the mobile phone When the mobile phone receives the call image P2, it obtains the mobile phone On the system time T2, if the difference between T2 and T1 is less than the set value (for example, 100ms), continue to interact with the tablet through the method shown in Figure 3 for multi-screen interaction, otherwise, the phone starts the screen projection method provided by this application , And send the instruction to start the screen projection method of this application to the tablet.
  • the set value for example, 100ms
  • whether to start the screen projection method of the present application is determined according to the amount of data buffer in the communication channel between the mobile phone and the tablet.
  • the amount of data buffering in the communication channel may be the amount of data buffering in the wireless communication module (for example, the wireless communication module 160 shown in FIG. 4) of the image data sending end.
  • the tablet sends user A’s call image P1 to the mobile phone, it detects the amount of data buffering in the tablet’s wireless communication module.
  • the tablet performs multi-screen interaction, otherwise, the tablet activates the screen projection method provided in this application, and sends an instruction to start the screen projection method of this application to the mobile phone.
  • the projection system 200 includes a projection source 210 (for example, a mobile phone) and a projection destination 220 (for example, a tablet).
  • a projection source 210 for example, a mobile phone
  • a projection destination 220 for example, a tablet
  • the screen projection source 210 includes:
  • the device virtualization module 211 includes a Display sub-module 211a.
  • the Display submodule 211a is configured to receive the user's first call image (for example, the call image P1 in step S10) sent by the projection destination 220, and transmit the received user's first call image to the video call application module 212;
  • the Display sub-module 211a is also used to obtain the image data of the interface element from the video call application module 212; and to send the image data of the interface element to the projection destination 220.
  • the interface elements include controls used to control the call state and/or the interface display state (e.g., element 1, element 2, element 3 in Figure 9a), and call state information (e.g., element 4 in Figure 9a) , User identification, and/or call images of other users participating in the video call (e.g., element 5 in Figure 9a).
  • the image data of the interface elements includes the image information of each interface element and the position information of each interface element on the video call interface displayed by the projection source 210.
  • the device virtualization module 211 may also determine whether to start the screen projection method provided in Embodiment 1 of the present application according to the communication state between the screen projection source 210 and the screen projection destination 220. For example, the device virtualization module 211 is based on the communication delay between the projection source 210 and the projection destination 220, and/or the amount of data buffer in the communication channel between the projection source 210 and the projection destination 220 , To determine whether to activate the projection method provided in this application.
  • the device virtualization module 211 determines to start the screen projection method provided in this application, and sends an instruction to start the screen projection method provided in this application to the screen projection destination 220.
  • video telephony application module 212 includes one or more video telephony application, e.g., micro-letters TM, Facetime TM, telephone systems and other applications.
  • the video call application module 212 is configured to receive the user's first call image from the projection destination 220 from the Display submodule 211a; and determine the interface elements of the video call interface displayed by the projection source 210 (for example, in FIG. 9a Element 1 to element 6); and, according to the received user's first call image and the determined interface elements, generate a video call interface for display at the screen source 210 (for example, the mobile phone side interface described in step S20);
  • the display module 213 is configured to display the video call interface generated by the video call application module 212.
  • the projection destination 220 includes:
  • the device virtualization module 221 includes a Display sub-module 221a.
  • the Display sub-module 221a sends the user's first call image (for example, the call image P1 in step S10) collected by the image capture module 222 to the projection source 220;
  • the Display sub-module 221a is also used to receive the image data of the interface elements sent by the projection source 210; and, to receive the user's second call image (for example, the call image P2 in step S40) collected in real time by the image collection module 222; and, According to the received interface elements and the user's second call image collected in real time by the image capture module 222, a video call interface (for example, the tablet-side interface described in step S50) displayed by the screen projection destination 220 is generated.
  • a video call interface for example, the tablet-side interface described in step S50
  • the device virtualization module 221 can also be used to obtain the current working status of the image acquisition module 222 and the display module 223 of the projection destination 220, for example, whether the camera in the image acquisition module 222 is in use, and whether the current working status of the display module 223 is Is the screen state;
  • the image acquisition module 222 including a camera, is used to collect the user's first call image; and, when the projection destination 220 receives the image data of the interface element from the projection source 210, it collects the user's second call in real time image;
  • the display module 223 includes a display screen for displaying the video call interface generated by the Display sub-module 221a.
  • the electronic device 400 may include one or more processors 401 coupled to the controller hub 403.
  • the controller hub 403 is connected via a multi-branch bus such as Front Side Bus (FSB), a point-to-point interface such as QuickPath Interconnect (QPI), or a similar connection 406 Communicate with the processor 401.
  • the processor 401 executes instructions that control general types of data processing operations.
  • the controller hub 403 includes, but is not limited to, a graphics memory controller hub (Graphics&Memory Controller Hub, GMCH) (not shown) and an input/output hub (Input Output Hub, IOH) (which may be On a separate chip) (not shown), where the GMCH includes a memory and a graphics controller and is coupled with the IOH.
  • GMCH Graphics&Memory Controller Hub
  • IOH Input Output Hub
  • the electronic device 400 may also include a coprocessor 402 and a memory 404 coupled to the controller hub 403.
  • a coprocessor 402 and a memory 404 coupled to the controller hub 403.
  • one or both of the memory and the GMCH may be integrated in the processor (as described in this application), and the memory 404 and the coprocessor 402 are directly coupled to the processor 401 and the controller hub 403, and the controller hub 403 and IOH are in a single chip.
  • the memory 404 may be, for example, a dynamic random access memory (Dynamic Random Access Memory, DRAM), a phase change memory (Phase Change Memory, PCM), or a combination of the two.
  • the memory 404 may include one or more tangible, non-transitory computer-readable media for storing data and/or instructions.
  • the computer-readable storage medium stores instructions, specifically, temporary and permanent copies of the instructions.
  • the electronic device 400 shown in FIG. 11 can be implemented as a screen projection source terminal and a screen projection destination terminal, respectively.
  • the instructions stored in the memory 404 may include: when executed by at least one of the processors, the projection source terminal causes the projection source terminal to implement the method shown in FIG. Instructions for the steps to be implemented.
  • the instructions stored in the memory 404 may include: when executed by at least one of the processors, the projection destination will cause the projection destination to implement the method shown in FIG. 7 by the projection destination. Instructions for the steps to be implemented.
  • the coprocessor 402 is a dedicated processor, such as, for example, a high-throughput Many Integrated Core (MIC) processor, a network or communication processor, a compression engine, a graphics processor, and a graphics processing unit.
  • MIC high-throughput Many Integrated Core
  • General-purpose computing General-purpose computing on graphics processing units, GPGPU), or embedded processors, etc.
  • the optional nature of the coprocessor 402 is shown in dashed lines in FIG. 11.
  • the electronic device 400 may further include a network interface (Network Interface Controller, NIC) 406.
  • the network interface 406 may include a transceiver, which is used to provide a radio interface for the electronic device 400 to communicate with any other suitable devices (such as a front-end module, an antenna, etc.).
  • the network interface 406 may be integrated with other components of the electronic device 400.
  • the network interface 406 can realize the function of the communication unit in the above-mentioned embodiment.
  • the electronic device 400 may further include an input/output (Input/Output, I/O) device 405.
  • I/O 405 may include: a user interface, which is designed to enable a user to interact with the electronic device 400; a peripheral component interface is designed to enable peripheral components to also interact with the electronic device 400; and/or a sensor is designed to determine the correlation with the electronic device 400 Environmental conditions and/or location information.
  • FIG. 11 is only exemplary. That is, although FIG. 11 shows that the electronic device 400 includes multiple devices such as the processor 401, the controller hub 403, and the memory 404, in actual applications, the devices using the methods of the present application may only include the electronic device 400. Some of the devices, for example, may only include the processor 401 and the network interface 406. The properties of optional devices in Fig. 11 are shown by dashed lines.
  • the SoC500 includes: an interconnection unit 550, which is coupled to the processor 510; a system agent unit 580; a bus controller unit 590; an integrated memory controller unit 540; a group or one or more coprocessors 520 , which may include integrated graphics logic, image processor, audio processor and video processor; Static Random-Access Memory (SRAM) unit 530; Direct Memory Access (DMA) unit 560 .
  • an interconnection unit 550 which is coupled to the processor 510; a system agent unit 580; a bus controller unit 590; an integrated memory controller unit 540; a group or one or more coprocessors 520 , which may include integrated graphics logic, image processor, audio processor and video processor; Static Random-Access Memory (SRAM) unit 530; Direct Memory Access (DMA) unit 560 .
  • SRAM Static Random-Access Memory
  • DMA Direct Memory Access
  • the coprocessor 520 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, general-purpose computing on graphics processing units (GPGPU), a high-throughput MIC Processor, or embedded processor, etc.
  • a dedicated processor such as, for example, a network or communication processor, a compression engine, general-purpose computing on graphics processing units (GPGPU), a high-throughput MIC Processor, or embedded processor, etc.
  • the static random access memory (SRAM) unit 530 may include one or more tangible, non-transitory computer-readable media for storing data and/or instructions.
  • the computer-readable storage medium stores instructions, specifically, temporary and permanent copies of the instructions.
  • the SoC as shown in Figure 12 can be set in the projection destination and the projection source respectively.
  • instructions are stored in the static random access memory (SRAM) unit 530.
  • the instructions may include: when executed by at least one of the processors, the wearable device will be implemented as shown in FIG. Instructions for the steps implemented by the projection destination in the method shown.
  • instructions are stored in the static random access memory (SRAM) unit 530.
  • the instructions may include: when executed by at least one of the processors, the projection source is implemented as shown in FIG. 7 Instructions for the steps implemented by the projection source in the method shown.
  • Program code can be applied to input instructions to perform the functions described in this article and generate output information.
  • the output information can be applied to one or more output devices in a known manner.
  • the processing system includes any processor having a processor such as, for example, a Digital Signal Processor (DSP), a microcontroller, an Application Specific Integrated Circuit (ASIC), or a microprocessor. system.
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • the program code can be implemented in a high-level programming language or an object-oriented programming language to communicate with the processing system.
  • assembly language or machine language can also be used to implement the program code.
  • the mechanisms described in this article are not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.
  • IP Intelligent Property
  • the instruction converter can be used to convert instructions from the source instruction set to the target instruction set.
  • the instruction converter may transform (for example, use static binary transformation, dynamic binary transformation including dynamic compilation), deform, emulate, or otherwise convert the instruction into one or more other instructions to be processed by the core.
  • the instruction converter can be implemented by software, hardware, firmware, or a combination thereof.
  • the instruction converter may be on the processor, off the processor, or part on the processor and part off the processor.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Telephone Function (AREA)

Abstract

一种投屏方法、投屏源端、投屏目的端、投屏系统及存储介质。该投屏方法应用于具有视频通话功能的投屏源端，方法包括：在视频通话状态下，接收投屏目的端发送的用户第一通话图像；生成用于投屏源端显示的视频通话界面，视频通话界面包括用户第一通话图像以及界面元素；向投屏目的端发送界面元素的图像数据，以使得投屏目的端将界面元素以及用户第二通话图像进行合成，以生成用于投屏目的端显示的界面；其中，用户第二通话图像为投屏目的端接收到界面元素的图像数据时，投屏目的端的摄像头采集的图像。该投屏方法能够降低视频通话过程中的用户感知时延。

Description

投屏方法、投屏源端、投屏目的端、投屏系统及存储介质
本申请要求2020年05月19日提交中国专利局、申请号为202010424716.X、申请名称为“投屏方法、投屏源端、投屏目的端、投屏系统及存储介质”的中国专利申请；以及2021年02月26日提交中国专利局、申请号为202110220988.2、申请名称为“投屏方法、投屏源端、投屏目的端、投屏系统及存储介质”的中国专利申请的优先权，上述申请的全部内容通过引用结合在本申请中。
技术领域
本申请涉及终端技术领域，尤其涉及一种投屏方法、投屏源端、投屏目的端、投屏系统以及计算机可读存储介质。
背景技术
随着终端设备的普及，用户往往会拥有多台终端设备(例如，可穿戴设备、手机、平板电脑等)，多屏互动技术可以为用户带来良好的使用体验。多屏互动技术是在不同的终端设备之间共享屏幕显示内容的技术，其中，视频通话(例如，视频聊天、多人视频会议等)界面的多屏共享是多屏互动技术的一个重要应用。
现有技术中，当用户通过多屏互动技术进行视频通话时，有可能感觉到视频通话界面显示的通话时延较为严重，这会影响用户体验。
发明内容
本申请的一些实施方式提供了一种投屏方法、投屏源端、投屏目的端和投屏系统，以下从多个方面介绍本申请，以下多个方面的实施方式和有益效果可互相参考。
第一方面，本申请实施方式提供了一种投屏方法，应用于具有视频通话功能的投屏源端，投屏方法包括：在视频通话状态下，接收投屏目的端发送的用户第一通话图像；生成用于投屏源端显示的视频通话界面，视频通话界面包括用户第一通话图像以及界面元素；向投屏目的端发送界面元素的图像数据，以使得投屏目的端将界面元素以及用户第二通话图像进行合成，以生成用于投屏目的端显示的界面；其中，用户第二通话图像为投屏目的端接收到界面元素的图像数据时，投屏目的端的摄像头采集的图像。
根据本申请的实施方式，投屏目的端将投屏源端的视频通话界面的界面元素与本地摄像头实时采集的用户通话图像进行合成，以生成用于投屏目的端显示的视频通话界面，这样，投屏目的端显示的视频通话界面上，用户通话图像的采集时间与显示时间仅相隔投屏目的端对图像数据进行处理的时间，从而可以显著降低用户感知时延。
在一些实施方式中，视频通话界面的界面元素包括下述至少一种：用于对通话状态和/或界面显示状态进行控制的控件、通话状态信息、用户标识、参与视频通话的其他用户的通话图像。
在一些实施方式中，界面元素的图像数据包括：界面元素中各个元素的图像信息，以及各个元素在视频通话界面上的位置信息。根据本申请的实施方式，可以减少界面元素的图像数据的数 据量。
在一些实施方式中，投屏方法的启动条件为：投屏源端与投屏目的端之间的通信时延超过设定阈值；和/或，投屏源端与投屏目的端之间的通信信道中的数据缓冲量超过设定阈值。
第二方面，本申请实施方式提供了一种投屏方法，应用于投屏目的端，投屏方法包括：当投屏源端处于视频通话状态时，向投屏源端发送用户第一通话图像，以使得投屏源端生成用于投屏源端显示的视频通话界面，视频通话界面包括用户第一通话图像以及界面元素；接收投屏源端发送的界面元素的图像数据；将界面元素以及用户第二通话图像进行合成，以生成用于投屏目的端显示的界面；其中，用户第二通话图像为投屏目的端接收到界面元素的图像数据时，投屏目的端的摄像头采集的图像。
根据本申请的实施方式，投屏目的端将投屏源端的视频通话界面的界面元素与本地摄像头实时采集的用户通话图像进行合成，以生成用于投屏目的端显示的视频通话界面，这样，投屏目的端显示的视频通话界面上，用户通话图像的采集时间与显示时间仅相隔投屏目的端对图像数据进行处理的时间，从而可以显著降低用户感知时延。
在一些实施方式中，视频通话界面的界面元素包括下述至少一种元素：用于对通话状态和/或界面显示状态进行控制的控件、通话状态信息、用户标识、参与视频通话的其他用户的通话图像。
在一些实施方式中，界面元素的图像数据包括：界面元素中各个元素的图像信息，以及各个元素在视频通话界面上的位置信息。根据本申请的实施方式，可以减少界面元素的图像数据的数据量。
在一些实施方式中，投屏方法的启动条件为：投屏源端与投屏目的端之间的通信时延超过设定阈值；和/或，投屏源端与投屏目的端之间的通信信道中的数据缓冲量超过设定阈值。
第三方面，本申请实施方式提供了一种投屏源端，投屏源端具有视频通话功能，投屏源端包括：存储器，用于存储由投屏源端的一个或多个处理器执行的指令；处理器，当处理器执行存储器中的指令时，可使得投屏源端执行本申请第一方面任一实施方式提供的投屏方法。第三方面能达到的有益效果可参考第一方面任一实施方式所提供的方法的有益效果，此处不再赘述。
第四方面，本申请实施方式提供了一种投屏目的端，投屏目的端具有视频通话功能，投屏目的端包括：存储器，用于存储由投屏目的端的一个或多个处理器执行的指令；处理器，当处理器执行存储器中的指令时，可使得投屏目的端执行本申请第二方面任一实施方式提供的投屏方法。第四方面能达到的有益效果可参考第二方面任一实施方式所提供的方法的有益效果，此处不再赘述。
第五方面，本申请实施方式提供了一种投屏系统，投屏系统包括投屏源端和投屏目的端，投屏源端具有视频通话功能，其中，投屏源端用于执行本申请第一方面任一实施方式提供的投屏方法，投屏目的端用于执行本申请第二方面任一实施方式提供的投屏方法。第五方面能达到的有益效果可参考第一方面任一实施方式或第二方面任一实施方式所提供的方法的有益效果，此处不再赘述。
第六方面，本申请实施方式提供一种计算机可读存储介质，计算机可读存储介质中存储有指令，该指令在计算机上执行时，可使计算机执行本申请第一方面任一实施方式提供的投屏方法，或者使计算机执行本申请第二方面任一实施方式提供的投屏方法。第六方面能达到的有益效果可参考第一方面任一实施方式或第二方面任一实施方式所提供的方法的有益效果，此处不再赘述。
附图说明
图1为本申请实施方式提供的投屏方法的示例性应用场景；
图2为本申请实施方式提供的图像数据传输过程示意图；
图3为现有技术中图像数据传输过程示意图；
图4为本申请实施方式提供的电子设备的构造示意图；
图5a为本申请实施方式提供的投屏源端的软件架构图；
图5b为本申请实施方式提供的投屏目的端的软件架构图；
图6a为本申请实施方式提供的多屏互动场景示意图(投屏源端)；
图6b为本申请实施方式提供的多屏互动场景示意图(投屏目的端)；
图7为本申请实施方式提供的投屏方法的流程示意图；
图8为本申请实施方式提供的手机侧界面示意图；
图9a为本申请实施方式提供的手机侧界面的界面元素图像数据示意图；
图9b为本申请实施方式提供的手机侧界面的界面元素图像数据另一个示意图；
图9c为本申请实施方式提供的平板侧界面示意图；
图10为本申请实施方式提供的投屏系统的示例性结构图；
图11示出了本申请实施方式提供的电子设备的框图；
图12示出了本申请实施方式提供的片上系统(SoC，System on Chip)的结构示意图。
具体实施方式
下面将结合附图对本实施例的实施方式进行详细描述。
图1示出了用户甲与用户乙进行视频通话的场景。
用户甲拥有设备A1和设备A2，其中，设备A1上安装有视频通话应用(例如，微信、skype等)，这样，在视频通话过程中，设备A1可通过视频通话应用生成视频通话界面。利用多屏互动技术，在视频通话过程中，设备A1可将视频通话界面的界面图像传输到设备A2上(例如，通过投屏技术进行传输，或者，基于分布式系统进行传输)，这样，设备A2可以显示设备A1生成的视频通话界面。通常情况下，视频通话界面包括用户甲的通话图像、用户乙的通话图像、界面控件、通话状态信息等。
有时，在视频通话过程中，设备A2为用户甲当前正在操作的设备(例如，用户甲正在使用设备A2上的某个应用(例如，图片处理应用)，或者，用户甲在驾驶车辆时需要通过设备A2(例如，车机)进行视频通话等)，此时，视频通话界面上用户甲的通话图像是由设备A2的摄像头采集的，设备A2的摄像头在采集到用户甲的通话图像后，将该通话图像传输到设备A1，设备A1基于设备A2传输过来的通话图像生成视频通话界面。之后，设备A1再将生成的视频通话界面的界面图像传输到设备A2上，使得正在操作设备A2的用户甲可以看到该视频通话界面。
本申请实施方式用于提供应用于图1所示的场景的一种投屏方法，用于降低用户感知时延。概述性地，本申请实施方式中，设备A2并非直接显示设备A1上的视频通话界面，而是将A1的视频通话界面中的界面元素图像与本地摄像头实时采集的用户通话图像进行合成，以生成用于设备A2显示的视频通话界面。这样，设备A2显示的视频通话界面中，用户通话图像是用户甲的实时图像，其与用户甲的当前状态基本是一致的，从而可以显著降低用户感知时延，提高用户体验。
本申请实施方式中，设备A1为投屏源端，设备A2为投屏目的端。为描述清楚，在下文中，将手机作为投屏源端的示例，将平板作为投屏目的端的示例，因此，将手机生成的视频通话界面称为“手机侧界面”，将平板生成的视频通话界面称为“平板侧界面”。
但可以理解，本申请不限于此，在其他示例中，投屏源端可以是其他具有视频通话功能的设备，例如，平板、笔记本电脑等；投屏目的端可以是其他具有显示功能的设备，例如，电视、车机等。需要说明的是，投屏源端和投屏目的端可以是相同类型的设备(例如，投屏源端和投屏目的端都是手机)，也可以是不同类型的设备(例如，投屏源端为手机，投屏目的端为车机)。
另外，本申请实施方式中，对参与视频通话的用户的数量不作限定，可以是一对一的视频通话，例如图1示出的用户甲与用户乙的视频通话场景；也可以是多对多的视频通话，例如，多方视频会议场景，线上教学场景等。
图2示出了本申请实施方式中视频通话过程中图像数据的传输过程。具体地，T1时刻，设备A2采集并向设备A1发送用户甲的通话图像P1；T2时刻，设备A1接收到设备A2发送的用户甲的通话图像P1，以及设备B发送的用户乙的通话图像；在T3时刻，设备A1向设备A2发送设备A1生成的视频通话界面(即，手机侧界面)的界面元素图像(其中不包含用户甲的通话图像P1)；T4’时刻，设备A2接收到设备A1发送的界面元素图像，并通过设备A2的摄像头实时采集用户甲的通话图像P2，然后将界面元素图像和用户甲的通话图像P2进行合成以生成用于设备A2显示的视频通话界面(即，平板侧界面)；T5’时刻，设备A2显示平板侧界面，平板侧界面上包括用户甲的通话图像P2。
图3示出了现有技术中已有的技术方案，图3与图2的主要区别是，T3时刻，设备A1基于用户甲的通话图像P1和用户乙的通话图像生成视频通话界面(即，手机侧界面)，并向设备A2发送手机侧界面的界面图像；T4时刻，设备A2接收到手机侧界面的界面图像；T5时刻，设备A2显示手机侧界面，该界面上包括设备A2在T1时刻采集的用户甲的通话图像P1。
假定，在T1时刻(即，设备A2采集用户甲的通话图像的时刻)，用户甲正在做抬手动作，经过时间间隔ΔT＝T5-T1之后，即在T5时刻，设备A2上会显示该抬手动作。当设备A2显示视频通话界面的时间相较于设备A2采集用户甲的通话图像的时间有较长的时延时，即，时间间隔ΔT较长时，用户甲会感知到设备A2显示的内容明显滞后于用户甲的当前状态，即，用户甲会感知到较为严重的通话时延(本文称为“用户感知时延”)，这样会影响用户甲的使用体验。
引起用户感知时延的原因主要是图像数据在通信网络中的传输时延，以及终端设备对图像数据进行处理的处理时延，其中，前者为引起用户感知时延的主要原因。在网络质量一般时，用户感知时延可能超过300ms，在网络比较拥堵时，用户感知时延可能会大于500ms，这样会降低用户的使用体验。
参考图2，本申请实施方式中，因为设备A2将设备A1的视频通话界面的界面元素与本地摄像头实时采集的用户通话图像进行合成以生成用于设备A2显示的视频通话界面，即，设备A2显示的视频通话界面上，用户甲的通话图像P2为设备A2的摄像头在T4’时刻采集到的图像，由于通话图像P2的采集时间(即，T4’)与显示时间(即，T5’)之间仅相隔设备A2对图像数据进行处理的时间，从而可以显著降低用户感知时延。
下文以图1示出的用户甲与用户乙的视频通话场景为例对本申请的技术方案进行介绍。
图4示出了电子设备100的结构示意图。其中，该电子设备100可以实现为手机，也可实现为平板。
电子设备100可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接头130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E，距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L，骨传导传感器180M等。
可以理解的是，本发明实施例示意的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中，电子设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
处理器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器110可以包含多组I2C总线。处理器110可以通过I2C总线接口耦合摄像头193。
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在 串行通信与 并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器110与无线通信模块160。例如：处理器110通过UART接口与无线通信模块160中的蓝牙模块通信，实现蓝牙功能。
MIPI接口可以被用于连接处理器110与显示屏194，摄像头193等外围器件。MIPI接口包括摄像头串行接口(camera serial interface，CSI)，显示屏串行接口(display serial interface，DSI)等。在一些实施例中，处理器110和摄像头193通过CSI接口通信，实现电子设备100的拍摄功能。处理器110和显示屏194通过DSI接口通信，实现电子设备100的显示功能。
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号，也可被配置为数据信号。在一些实施例中，GPIO接口可以用于连接处理器110与摄像头193，显示屏194，无线通信模块 160，音频模块170等。GPIO接口还可以被配置为I2C接口，UART接口，MIPI接口等。
可以理解的是，本发明实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对电子设备100的结构限定。在本申请另一些实施例中，电子设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。
电子设备100的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调处理器以及基带处理器等实现。
天线1和天线2用于发射和接收电磁波信号。电子设备100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
移动通信模块150可以提供应用在电子设备100上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器170A，受话器170B等)输出声音信号，或通过显示屏194显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块150或其他功能模块设置在同一个器件中。
无线通信模块160可以提供应用在电子设备100上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。
在一些实施例中，电子设备100的天线1和移动通信模块150耦合，天线2和无线通信模块160耦合，使得电子设备100可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigation satellite system，GLONASS)，北斗卫星导航系统(beidou navigation satellite system，BDS)，准天顶卫星系统(quasi-zenith satellite system，QZSS)和/或星基 增强系统(satellite based augmentation systems，SBAS)。
电子设备100可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。
音频模块170用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块170还可以用于对音频信号编码和解码。在一些实施例中，音频模块170可以设置于处理器110中，或将音频模块170的部分功能模块设置于处理器110中。
电子设备100通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，电子设备100可以包括1个或N个显示屏194，N为大于1的正整数。
电子设备100可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头193中。
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，电子设备100可以包括1个或N个摄像头193，N为大于1的正整数。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当电子设备100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。电子设备100可以支持一种或多种视频编解码器。这样，电子设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现电子设备100的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
外部存储器接口120可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备100的存储能力。外部存储卡通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。内部 存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储电子设备100使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。内部存储器121和/或设置于处理器中的存储器存储有指令。当电子设备100被实现为手机时，内部存储器121和/或设置于处理器中的存储器存储的指令由处理器110执行时，可使手机执行本申请实施方式提供的投屏方法中由手机执行的步骤；当电子设备100被实现为平板时，内部存储器121和/或设置于处理器中的存储器存储的指令由处理器110执行时，可使平板执行本申请实施方式提供的投屏方法中由平板执行的步骤。
应理解，在实际应用中，电子设备100可以包括比图4所示的更多或更少的部件，本申请实施例不作限定。图示电子设备100仅是一个范例，并且设备100可以具有比图中所示出的更多的或者更少的部件，可以组合两个或更多的部件，或者可以具有不同的部件配置。图中所示出的各种部件可以在包括一个或多个信号处理和/或专用集成电路在内的硬件、软件、或硬件和软件的组合中实现。
以下介绍手机的软件系统。手机的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本发明实施例以分层架构的Android系统为例，示例性说明手机系统架构的软件结构。
图5a示出了手机的系统架构。分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android系统分为四层，从上至下分别为应用程序层，应用程序框架层，安卓系统运行时(Android runtime)和系统库，以及内核层。
应用程序层可以包括一系列应用程序包。如图5a所示，手机上的应用程序包可以包括视频通话应用、手机投屏应用、相机，图库，日历，通话，地图，导航，WLAN，蓝牙，音乐，视频，短信息等应用程序。
视频通话应用用于在视频通话过程中生成手机侧界面。具体地，在视频通话过程中，视频通话应用获取用户甲的通话图像以及用户乙的通话图像，并生成通话控件(例如，对通话状态进行控制的控件，对界面显示状态进行控制的控件)、用户标识等，之后，视频通话应用将用户甲的通话图像、用户乙的通话图像、通话控件、用户标识等进行合成，以形成手机侧界面。本申请实施方式中，将手机侧界面上除用户甲的通话图像之外的其他元素(例如，用户乙的通话图像，通话控件等)称为界面元素。
视频通话应用还将手机侧界面的界面元素的图像数据传输至手机投屏应用。手机投屏应用在接收到界面元素的图像数据之后，对界面元素的图像数据进行处理，例如，编码、封装等，然后手机将处理后的图像数据发送至平板。
如图5a所示，应用程序框架层可以包括窗口管理器，内容提供器，视图系统，电话管理器，资源管理器，通知管理器等。
窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏，锁定屏幕，截取屏幕等。
内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。
视图系统包括可视控件，例如显示文字的控件，显示图片的控件等。视图系统可用于构建应 用程序。显示界面可以由一个或多个视图组成的。例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。
电话管理器用于提供手机的通信功能。例如通话状态的管理(包括接通，挂断等)。
资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。
通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于告知下载完成，消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，发出提示音，电子设备振动，指示灯闪烁等。
Android Runtime包括核心库和虚拟机。Android runtime负责安卓系统的调度和管理。
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程管理，安全和异常的管理，以及垃圾回收等功能。
系统库可以包括多个功能模块。例如：表面管理器(surface manager)，媒体库(Media Libraries)，三维图形处理库(例如：OpenGL ES)，2D图形引擎(例如：SGL)等。
表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D和3D图层的融合。
媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如:MPEG4，H.264，MP3，AAC，AMR，JPG，PNG等。
三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。
2D图形引擎是2D绘图的绘图引擎。
内核层是硬件和软件之间的层。内核层至少包含显示驱动，摄像头驱动，音频驱动，传感器驱动等。
图5b示出了平板的系统架构。平板的系统架构与手机的系统架构基本相同，下面仅介绍平板系统架构中与手机系统架构相区别的部分，对于两者的相同部分不再赘述。
平板的应用程序层包括平板投屏应用和手机分身应用。
平板接收到手机发送的手机侧界面的界面元素图像数据后，平板投屏应用对接收到的图像数据进行解码，以还原出手机侧界面的界面元素图像。平板侧投屏还可以获取平板摄像头采集的用户通话图像P2，并将平板侧界面的界面元素图像与用户通话图像P2进行合成，以生成叠加图像。
手机分身应用可以获取平板投屏应用生成的叠加图像，并对叠加图像进行处理，例如，对叠加图像的分辨率进行调整，在叠加图像上添加投屏源端(即，手机)的标识信息等，以生成平板侧界面。
以下结合图1示出的多屏互动场景对本申请的具体实施例进行介绍。
在本实施例的方法开始之前，用户甲的手机和平板处于多屏互动状态，即，手机通过多屏互动技术将其屏幕显示内容共享至平板上，平板通过其安装的手机分身应用显示手机屏幕的内容。示例性地，参考图6a和图6b，平板上的手机分身应用显示手机上的应用程序列表。在该状态下，视频通话的一方向另一方发起通话视频通话请求。
以用户甲向用户乙发起视频通话请求为例。参考图6b，用户甲在平板上点击视频通话应用的 图标之后，平板将用户甲的操作信息(例如，操作位置、操作类型等)发送至手机，手机根据平板发送的操作信息，确定用户的指令为开启视频通话应用。响应于接收到的用户指令，手机开启视频通话应用，并显示可建立视频通话连接的联系人界面。
平板同步显示视频通话应用的联系人界面。用户甲点击用户乙头像后，可使得手机向用户乙的设备(即，设备B)发送视频通话请求。用户乙在向其视频通话设备发出接受视频通话请求的指令后，用户甲的手机与用户乙的设备之间建立视频通话连接。
为降低视频通话状态下的用户感知时延，参考图7，本实施例用于提供一种视频通话状态下的投屏方法，包括以下步骤：
S10：在视频通话状态下，手机获取用户甲的通话图像P1(作为用户第一通话图像)以及用户乙的通话图像，其中，用户甲的通话图像是平板摄像头采集的图像，用户乙的通话图像为用户乙的设备采集的图像。
当手机与用户乙的设备建立视频通话连接后，手机向平板发送通话图像获取请求(例如，手机向平板发送视频通话连接建立的信号)。响应于手机发送的通话图像获取请求，平板启动摄像头，并以一定的帧率(例如，40fps)拍摄用户甲的通话图像。在当前步骤中，平板将摄像头拍摄到的用户甲的通话图像P1发送至手机。
同时，用户乙的设备采集用户乙的通话图像，并将用户乙的通话图像发送至手机。
需要说明的是，本实施例中，用户的通话图像为设备的摄像头拍摄到的图像，并不一定是用户本人的图像。例如，当用户甲进行线上教学时，平板的摄像头朝向演示板的位置，此时，用户甲的通话图像为演示板的画面。
另外，平板还可以采集用户甲的语音信号，并将用户甲的语音信号发送至手机，手机通过视频通话应用将用户甲的语音信号发送至用户乙的设备，以使得用户乙可以听到用户甲的声音。同样地，用户乙的设备采集用户乙的语音信号，并将用户乙的语音信号发送至手机，手机将用户乙的语音信号发送还平板，以使得用户甲可以听到用户乙的声音。
S20：手机生成手机侧界面(作为用于投屏源端显示的视频通话界面)，本实施例中，手机通过视频通话应用生成手机侧界面。具体地，手机视频应用确定用于手机显示的视频通话界面上的通话控件、通话信息等，并将通话控件、通话信息等与用户甲的通话图像P1、用户乙的通话图像进行合成，以生成手机侧界面。本申请中，将手机侧界面上除用户甲的通话图像之外的元素称为手机侧界面的“界面元素”。
参考图8，本实施例中，手机侧界面的界面元素包括下述一个或多个：用于对通话状态进行控制的控件，例如，挂断按钮10；语音通话/视频通话切换按钮20等；用于对界面显示状态进行控制的控件，例如，前置摄像头/后置摄像头切换按钮30，通话图像切换按钮(用于切换用户甲通话图像与用户乙通话图像的显示位置)等；通话状态信息，例如，通话时长40等；用户乙的通话图像50(作为参与视频通话的其他用户的通话图像)等。应理解，在其他实施例中，界面元素可以包括比本实施例更多或更少的元素，例如，在一些实施例中，界面元素还可以包括用户标识(例如，用户甲和/或用户乙的头像和昵称等)、界面边框等。
本实施例中，用户甲的通话图像P1显示于手机侧界面的右上角，用户乙的通话图像则分布于整个手机侧界面上。可以理解，在其他实施例中，用户的通话图像和用户乙的通话图像可以有不同的分布方式，例如，用户甲的通话图像和用户乙的通话图像按照图8的设置对换位置，或者，用户甲的通话图像和用户乙的通话图像并排显示于手机侧界面上。
S30：手机将界面元素的图像数据发送至平板。本实施例中，手机的视频通话应用确定手机侧界面的界面元素之后，将界面元素的图像数据传输至手机投屏应用，手机投屏应用对界面元素的图像数据进行处理后，再将处理后的图像数据发送至平板。
本实施例中，界面元素的图像数据包括各界面元素的图像信息以及各界面元素在手机侧界面上的位置信息。参考图9a，为便于表述，对视频通话界面上的界面元素进行编号，具体地，界面元素被分别编号为元素1、元素2，……，元素6。以元素2(即，用于控制视频通话通断状态的开关控件)为例，元素2的图像信息为形成元素2的各像素的像素值，元素2的位置信息为元素2在手机侧界面上的坐标。例如，在以图9a所示方式建立的坐标系xOy中，元素2的位置信息是其中心点的坐标(35mm，10mm)。
本实施例中，界面元素的图像数据包括各界面元素的图像信息和位置信息，这样，界面元素的图像数据的数据量相对较少。但本申请不限于此，例如，在其他实施例中，参考图9b，界面元素的图像数据为虚框区域中各像素的像素值。
手机视频通话应用确定界面元素的图像数据后，将图像数据发送至手机投屏应用，手机投屏应用对图像数据进行处理，例如，对图像数据进行编码、按照投屏协议(例如，miracast协议、airplay镜像协议等)进行封装等，之后将处理后的界面元素图像数据发送至平板。
S40：平板生成用户甲的通话图像P2(作为用户第二通话图像)与界面元素图像的叠加图像。本实施例中，步骤S40具体包括：
S41：平板投屏应用还原出界面元素图像。平板在接收到手机发送的界面元素的图像数据后，通过平板投屏应用对图像数据进行解封装、解码之后，还原出各界面元素的图像数据，并根据界面元素的图像数据还原出界面元素图像(即，手机侧界面中除用户通话图像P1之外的其他部分的图像)。
S42：平板投屏应用获取用户甲的通话图像P2。平板在接收到手机发送的界面元素的图像数据后，平板摄像头实时采集用户甲的通话图像P2，平板投屏应用通过平板底层系统获取用户甲的通话图像P2。
S43：平板投屏应用将用户甲的通话图像P2与界面元素图像进行叠加，以生成叠加图像。本实施例中，平板根据手机侧界面上用户甲的通话图像P1的大小和位置(该信息可以是手机发送给平板的)，确定用户甲的通话图像P2在叠加图像上的大小和位置，以使得叠加图像与手机侧界面尽可能接近。但本申请不限于此，例如，在其他实施例中，用户甲的通话图像P2的大小及其在叠加图像上的位置是用户设定的。
S50：平板根据叠加图像生成平板侧界面(作为用于投屏目的端显示的界面)。本实施例中，平板投屏应用将生成的叠加图像传输至平板上的手机分身应用，手机分身应用对叠加图像进行处理，例如，对叠加图像的分辨率进行调整，以使平板侧界面的面积与平板显示屏的面积相匹配；和/或，在叠加图像上添加投屏源端的标识信息(例如，在叠加图像上添加文字“来自Mate40的投屏”)等，以生成平板侧界面。之后，手机分身应用将平板侧界面的界面图像传送至平板底层系统，以使得平板底层系统控制平板显示屏显示平板侧界面(如图9c所示)。
需要说明的是，本实施例的技术方案仅为本申请实施方式的示例性说明，本申请可以有其他的变形方式。
例如，可以对本实施例中的各应用进行合并、分拆或重新组合。例如，可以将手机上的视频通话应用和手机投屏应用可以合并为一个应用；或者，将平板投屏应用的部分功能(例如，步骤 S43中的图像叠加功能)合并到平板上的手机分身应用。
再如，还可以省去本实施例中的某些步骤，或在本实施例的基础上增加步骤。例如，省去步骤S30中手机投屏模块对界面元素的图像数据进行处理的步骤，手机直接将视频通话应用确定的界面元素的图像数据发送至平板；或者，省去步骤S50中手机分身应用对叠加图像进行处理的步骤，平板直接将平板投屏应用生成的叠加图像作为平板侧界面进行显示等。
另外，还可以对本申请提供的投屏方法设置启动条件。当手机和平板所在的通信网络畅通时，可采用现有技术中的方法(例如，图3所示的方法)进行多屏互动。当通信网络较为拥堵时，可启动本申请提供的投屏方法，以降低用户感知时延。
在一些实施例中，根据平板与手机之间的通信时延确定是否启动本申请提供的投屏方法。示例性地，通信时延可以根据平板与手机之间的图像传输时间进行确定。例如，参考图2，平板向手机发送的用户甲的通话图像P1中带有时间戳，手机通过读取时间戳可获取通话图像P1的发送时间T1，手机在接收到通话图像P2时，获取手机上的系统时间T2，如果T2与T1的差值小于设定值(例如，100ms)，则继续通过图3所示的方法与平板进行多屏互动，否则，手机启动本申请提供的投屏方法，并向平板发送启动本申请投屏方法的指令。
在另一些实施例中，根据手机与平板之间的通信信道中的数据缓冲量确定是否启动本申请的投屏方法。示例性地，通信信道中数据缓冲量可以是图像数据发送端的无线通信模块(例如，图4所示的无线通信模块160)中的数据缓冲量。例如，继续参考图3，当平板向手机发送用户甲的通话图像P1时，检测平板无线通信模块中的数据缓冲量，如果缓冲量小于设定值时，则继续通过图2所示的方法与平板进行多屏互动，否则，平板启动本申请提供的投屏方法，并向手机发送启动本申请投屏方法的指令。
在又一些实施例中，根据平板与手机之间的通信时延，以及平板与手机之间通信信道中的数据缓冲量综合判断是否启动本申请的投屏方法。例如，当平板与手机之间的通信时延超过设定值，或手机与平板之间的通信信道中的数据缓冲量超过设定值时，启动本申请提供的投屏方法。
以下结合图10说明根据本申请实施例提供的投屏系统200的结构示意图。参考图10，投屏系统200包括投屏源端210(例如，手机)和投屏目的端220(例如，平板)。
其中，投屏源端210包括：
(1)设备虚拟化模块211，设备虚拟化模块211包括Display子模块211a。
Display子模块211a用于接收投屏目的端220发送的用户第一通话图像(例如，步骤S10中的通话图像P1)，并将接收到的用户第一通话图像传输至视频通话应用模块212；
Display子模块211a还用于从视频通话应用模块212中获取界面元素的图像数据；以及，向投屏目的端220发送界面元素的图像数据。其中，界面元素包括用于对通话状态和/或界面显示状态进行控制的控件(例如，图9a中的元素1、元素2、元素3)，通话状态信息(例如，图9a中的元素4)，用户标识，和/或参与视频通话的其他用户的通话图像(例如，图9a中的元素5)。界面元素的图像数据包括各界面元素的图像信息以及各界面元素在投屏源端210显示的视频通话界面上的位置信息。
设备虚拟化模块211还可以根据投屏源端210和投屏目的端220之间的通信状态，判断是否启动本申请实施例一提供的投屏方法。例如，设备虚拟化模块211根据投屏源端210与投屏目的端220之间的通信时延，和/或投屏源端210与投屏目的端220之间的通信信道中的数据缓冲量，判断是否启动本申请提供的投屏方法。具体地，当投屏源端210与投屏目的端220之间的通信时 延大于设定阈值，或者，投屏源端210与投屏目的端220之间的通信信道中的数据缓冲量大于设定阈值时，设备虚拟化模块211确定启动本申请提供的投屏方法，并向投屏目的端220发送启动本申请提供的投屏方法的指令。
(2)视频通话应用模块212，包括一个或多个视频通话应用，例如，微信 TM，Facetime TM，系统电话应用等。视频通话应用模块212用于从Display子模块211a接收来自投屏目的端220的用户第一通话图像；以及，确定用于投屏源端210显示的视频通话界面的界面元素(例如，图9a中的元素1～元素6)；以及，根据接收到的用户第一通话图像和确定的界面元素生成用于投屏源端210显示的视频通话界面(例如，步骤S20所述的手机侧界面)；
(3)显示模块213，用于显示视频通话应用模块212生成的视频通话界面。
投屏目的端220包括：
(1)设备虚拟化模块221，包括Display子模块221a。
Display子模块221a将图像采集模块222采集到的用户第一通话图像(例如，步骤S10中的通话图像P1)发送至投屏源端220；
Display子模块221a还用于接收投屏源端210发送的界面元素的图像数据；以及，接收图像采集模块222实时采集的用户第二通话图像(例如，步骤S40中的通话图像P2)；以及，根据接收到的界面元素和图像采集模块222实时采集的用户第二通话图像，生成用于投屏目的端220显示的视频通话界面(例如，步骤S50所述的平板侧界面)。
设备虚拟化模块221还可以用于获取投屏目的端220的图像采集模块222和显示模块223的当前工作状态，例如，图像采集模块222中的摄像头是否正在使用，显示模块223的当前工作状态是否是投屏状态；
(2)图像采集模块222，包括摄像头，用于采集用户第一通话图像；以及，在投屏目的端220接收到来自投屏源端210的界面元素的图像数据时，实时采集用户第二通话图像；
(3)显示模块223，包括显示屏，用于显示Display子模块221a生成的视频通话界面。
现在参考图11，所示为根据本申请的一个实施例的电子设备400的框图。电子设备400可以包括耦合到控制器中枢403的一个或多个处理器401。对于至少一个实施例，控制器中枢403经由诸如前端总线(Front Side Bus，FSB)之类的多分支总线、诸如快速通道互连(QuickPath Interconnect，QPI)之类的点对点接口、或者类似的连接406与处理器401进行通信。处理器401执行控制一般类型的数据处理操作的指令。在一实施例中，控制器中枢403包括，但不局限于，图形存储器控制器中枢(Graphics&Memory Controller Hub，GMCH)(未示出)和输入/输出中枢(Input Output Hub，IOH)(其可以在分开的芯片上)(未示出)，其中GMCH包括存储器和图形控制器并与IOH耦合。
电子设备400还可包括耦合到控制器中枢403的协处理器402和存储器404。或者，存储器和GMCH中的一个或两者可以被集成在处理器内(如本申请中所描述的)，存储器404和协处理器402直接耦合到处理器401以及控制器中枢403，控制器中枢403与IOH处于单个芯片中。
存储器404可以是例如动态随机存取存储器(Dynamic Random Access Memory，DRAM)、相变存储器(Phase Change Memory，PCM)或这两者的组合。存储器404中可以包括用于存储数据和/或指令的一个或多个有形的、非暂时性计算机可读介质。计算机可读存储介质中存储有指令，具体而言，存储有该指令的暂时和永久副本。
如图11所示的电子设备400可以被分别被实现为投屏源端和投屏目的端。当电子设备400被 实现为投屏源端时，存储器404中存储的指令可以包括：由处理器中的至少一个执行时导致投屏源端实施如图7所示的方法中由投屏源端实施的步骤的指令。当电子设备400被实现为投屏目的端时，存储器404中存储的指令可以包括：由处理器中的至少一个执行时致使投屏目的端实施如图7所示的方法中由投屏目的端实施的步骤的指令。
在一个实施例中，协处理器402是专用处理器，诸如例如高吞吐量集成众核(Many Integrated Core，MIC)处理器、网络或通信处理器、压缩引擎、图形处理器、图形处理单元上的通用计算(General-purpose computing on graphics processing units，GPGPU)、或嵌入式处理器等等。协处理器402的任选性质用虚线表示在图11中。
在一个实施例中，电子设备400可以进一步包括网络接口(Network Interface Controller，NIC)406。网络接口406可以包括收发器，用于为电子设备400提供无线电接口，进而与任何其他合适的设备(如前端模块，天线等)进行通信。在各种实施例中，网络接口406可以与电子设备400的其他组件集成。网络接口406可以实现上述实施例中的通信单元的功能。
电子设备400可以进一步包括输入/输出(Input/Output，I/O)设备405。I/O405可以包括：用户界面，该设计使得用户能够与电子设备400进行交互；外围组件接口的设计使得外围组件也能够与电子设备400交互；和/或传感器设计用于确定与电子设备400相关的环境条件和/或位置信息。
值得注意的是，图11仅是示例性的。即虽然图11中示出了电子设备400包括处理器401、控制器中枢403、存储器404等多个器件，但是，在实际的应用中，使用本申请各方法的设备，可以仅包括电子设备400各器件中的一部分器件，例如，可以仅包含处理器401和网络接口406。图11中可选器件的性质用虚线示出。
现在参考图12，所示为根据本申请的一实施例的片上系统(System on Chip，SoC)500的框图。在图12中，相似的部件具有同样的附图标记。另外，虚线框是更先进的SoC的可选特征。在图12中，SoC500包括：互连单元550，其被耦合至处理器510；系统代理单元580；总线控制器单元590；集成存储器控制器单元540；一组或一个或多个协处理器520，其可包括集成图形逻辑、图像处理器、音频处理器和视频处理器；静态随机存取存储器(Static Random-Access Memory，SRAM)单元530；直接存储器存取(Direct Memory Access，DMA)单元560。在一个实施例中，协处理器520包括专用处理器，诸如例如网络或通信处理器、压缩引擎、图形处理单元上的通用计算(General-purpose computing on graphics processing units，GPGPU)、高吞吐量MIC处理器、或嵌入式处理器等。
静态随机存取存储器(SRAM)单元530可以包括用于存储数据和/或指令的一个或多个有形的、非暂时性计算机可读介质。计算机可读存储介质中存储有指令，具体而言，存储有该指令的暂时和永久副本。
如图12所示的SoC可以被分别设置在投屏目的端和投屏源端中。当SoC被设置在投屏目的端中时，静态随机存取存储器(SRAM)单元530中存储有指令，该指令可以包括：由处理器中的至少一个执行时导致可穿戴设备实施如图7所示的方法中由投屏目的端实施的步骤的指令。当SoC被设置在投屏源端中时，静态随机存取存储器(SRAM)单元530中存储有指令，该指令可以包括：由处理器中的至少一个执行时致使投屏源端实施如图7所示的方法中由投屏源端实施的步骤的指令。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如， A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。
本申请的各方法实施方式均可以以软件、磁件、固件等方式实现。
可将程序代码应用于输入指令，以执行本文描述的各功能并生成输出信息。可以按已知方式将输出信息应用于一个或多个输出设备。为了本申请的目的，处理系统包括具有诸如例如数字信号处理器(Digital Signal Processor，DSP)、微控制器、专用集成电路(Application Specific Integrated Circuit，ASIC)或微处理器之类的处理器的任何系统。
程序代码可以用高级程序化语言或面向对象的编程语言来实现，以便与处理系统通信。在需要时，也可用汇编语言或机器语言来实现程序代码。事实上，本文中描述的机制不限于任何特定编程语言的范围。在任一情形下，该语言可以是编译语言或解释语言。
至少一个实施例的一个或多个方面可以由存储在计算机可读存储介质上的表示性指令来实现，指令表示处理器中的各种逻辑，指令在被机器读取时使得该机器制作用于执行本文所述的技术的逻辑。被称为“知识产权(Intellectual Property，IP)核”的这些表示可以被存储在有形的计算机可读存储介质上，并被提供给多个客户或生产设施以加载到实际制造该逻辑或处理器的制造机器中。
在一些情况下，指令转换器可用来将指令从源指令集转换至目标指令集。例如，指令转换器可以变换(例如使用静态二进制变换、包括动态编译的动态二进制变换)、变形、仿真或以其它方式将指令转换成将由核来处理的一个或多个其它指令。指令转换器可以用软件、硬件、固件、或其组合实现。指令转换器可以在处理器上、在处理器外、或者部分在处理器上且部分在处理器外。

Claims (12)

  1. 一种投屏方法，其特征在于，应用于具有视频通话功能的投屏源端，所述投屏方法包括：
    在视频通话状态下，接收投屏目的端发送的用户第一通话图像；
    生成用于所述投屏源端显示的视频通话界面，所述视频通话界面包括所述用户第一通话图像以及界面元素；
    向所述投屏目的端发送所述界面元素的图像数据，以使得所述投屏目的端将所述界面元素以及用户第二通话图像进行合成，以生成用于所述投屏目的端显示的界面；其中，所述用户第二通话图像为所述投屏目的端接收到所述界面元素的图像数据时，所述投屏目的端的摄像头采集的图像。
  2. 根据权利要求1所述的投屏方法，其特征在于，所述视频通话界面的界面元素包括下述至少一种：用于对通话状态和/或界面显示状态进行控制的控件、通话状态信息、用户标识、参与视频通话的其他用户的通话图像。
  3. 根据权利要求2所述的投屏方法，其特征在于，所述界面元素的图像数据包括：所述界面元素中各个元素的图像信息，以及各个元素在所述视频通话界面上的位置信息。
  4. 根据权利要求1所述的投屏方法，其特征在于，所述投屏方法的启动条件为：
    所述投屏源端与所述投屏目的端之间的通信时延超过设定阈值；和/或，
    所述投屏源端与所述投屏目的端之间的通信信道中的数据缓冲量超过设定阈值。
  5. 一种投屏方法，其特征在于，应用于投屏目的端，所述投屏方法包括：
    当投屏源端处于视频通话状态时，向所述投屏源端发送用户第一通话图像，以使得所述投屏源端生成用于所述投屏源端显示的视频通话界面，所述视频通话界面包括所述用户第一通话图像以及界面元素；
    接收所述投屏源端发送的所述界面元素的图像数据；
    将所述界面元素以及用户第二通话图像进行合成，以生成用于所述投屏目的端显示的界面；其中，所述用户第二通话图像为所述投屏目的端接收到所述界面元素的图像数据时，所述投屏目的端的摄像头采集的图像。
  6. 根据权利要求5所述的投屏方法，其特征在于，所述视频通话界面的界面元素包括下述至少一种元素：用于对通话状态和/或界面显示状态进行控制的控件、通话状态信息、用户标识、参与视频通话的其他用户的通话图像。
  7. 根据权利要求6所述的投屏方法，其特征在于，所述界面元素的图像数据包括：所述界面元素中各个元素的图像信息，以及各个元素在所述视频通话界面上的位置信息。
  8. 根据权利要求5所述的投屏方法，其特征在于，所述投屏方法的启动条件为：
    所述投屏源端与所述投屏目的端之间的通信时延超过设定阈值；和/或，
    所述投屏源端与所述投屏目的端之间的通信信道中的数据缓冲量超过设定阈值。
  9. 一种投屏源端，其特征在于，所述投屏源端具有视频通话功能，所述投屏源端包括：
    存储器，用于存储由所述投屏源端的一个或多个处理器执行的指令；
    处理器，当所述处理器执行所述存储器中的所述指令时，可使得所述投屏源端执行权利要求1～4任一项所述的投屏方法。
  10. 一种投屏目的端，其特征在于，包括：
    存储器，用于存储由所述投屏目的端的一个或多个处理器执行的指令；
    处理器，当所述处理器执行所述存储器中的所述指令时，可使得所述投屏目的端执行权利要求5～8任一所述的投屏方法。
  11. 一种投屏系统，其特征在于，所述投屏系统包括投屏源端和投屏目的端，所述投屏源端具有视频通话功能，其中，所述投屏源端用于执行如权利要求1～4任一所述的投屏方法，所述投屏目的端用于执行如权利要求5～8任一所述的投屏方法。
  12. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有指令，该指令在计算机上执行时，可使所述计算机执行权利要求1～4任一所述的投屏方法，或者使所述计算机执行权利要求5～8任一所述的投屏方法。
PCT/CN2021/093768 2020-05-19 2021-05-14 投屏方法、投屏源端、投屏目的端、投屏系统及存储介质 WO2021233218A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202010424716 2020-05-19
CN202010424716.X 2020-05-19
CN202110220988.2 2021-02-26
CN202110220988.2A CN113687803A (zh) 2020-05-19 2021-02-26 投屏方法、投屏源端、投屏目的端、投屏系统及存储介质

Publications (1)

Publication Number Publication Date
WO2021233218A1 true WO2021233218A1 (zh) 2021-11-25

Family

ID=78576280

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/093768 WO2021233218A1 (zh) 2020-05-19 2021-05-14 投屏方法、投屏源端、投屏目的端、投屏系统及存储介质

Country Status (2)

Country Link
CN (1) CN113687803A (zh)
WO (1) WO2021233218A1 (zh)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114786060A (zh) * 2022-02-28 2022-07-22 长沙朗源电子科技有限公司 基于安卓平台无线投屏实现帧率控制的方法及其装置
CN115119032A (zh) * 2022-06-22 2022-09-27 北斗星通智联科技有限责任公司 一种可游戏投屏的汽车智能座舱及方法
CN116033209A (zh) * 2022-08-29 2023-04-28 荣耀终端有限公司 投屏方法和电子设备
CN116055623A (zh) * 2022-06-23 2023-05-02 荣耀终端有限公司 一种功耗控制方法及电子设备
CN116737289A (zh) * 2022-09-15 2023-09-12 荣耀终端有限公司 显示方法、终端设备及存储介质
WO2023169276A1 (zh) * 2022-03-11 2023-09-14 华为技术有限公司 投屏方法、终端设备及计算机可读存储介质

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116367214A (zh) * 2021-12-28 2023-06-30 荣耀终端有限公司 通话状态监听方法、设备、存储介质
CN115022982B (zh) * 2021-12-31 2023-04-11 荣耀终端有限公司 多屏协同无感接入方法、电子设备及存储介质
CN114501089B (zh) * 2022-01-30 2023-05-05 深圳创维－Rgb电子有限公司 投屏通话方法、装置、电子设备及存储介质
CN117156189A (zh) * 2023-02-27 2023-12-01 荣耀终端有限公司 投屏显示方法及电子设备
CN115964011B (zh) * 2023-03-16 2023-06-06 深圳市湘凡科技有限公司 基于多屏协同的显示应用界面的方法及相关装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150091891A1 (en) * 2013-09-30 2015-04-02 Dumedia, Inc. System and method for non-holographic teleportation
US20170193182A1 (en) * 2015-12-31 2017-07-06 Dan M. MIHAI Distributed Telemedicine System and Method
CN107527623A (zh) * 2017-08-07 2017-12-29 广州视源电子科技股份有限公司 传屏方法、装置、电子设备及计算机可读存储介质
CN109218731A (zh) * 2017-06-30 2019-01-15 腾讯科技（深圳）有限公司 移动设备的投屏方法、装置及系统
CN110109636A (zh) * 2019-04-28 2019-08-09 华为技术有限公司 投屏方法、电子设备以及系统
CN110377256A (zh) * 2019-07-16 2019-10-25 浙江大华技术股份有限公司 多媒体文件的投放方法及装置、存储介质、电子装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150091891A1 (en) * 2013-09-30 2015-04-02 Dumedia, Inc. System and method for non-holographic teleportation
US20170193182A1 (en) * 2015-12-31 2017-07-06 Dan M. MIHAI Distributed Telemedicine System and Method
CN109218731A (zh) * 2017-06-30 2019-01-15 腾讯科技（深圳）有限公司 移动设备的投屏方法、装置及系统
CN107527623A (zh) * 2017-08-07 2017-12-29 广州视源电子科技股份有限公司 传屏方法、装置、电子设备及计算机可读存储介质
CN110109636A (zh) * 2019-04-28 2019-08-09 华为技术有限公司 投屏方法、电子设备以及系统
CN110377256A (zh) * 2019-07-16 2019-10-25 浙江大华技术股份有限公司 多媒体文件的投放方法及装置、存储介质、电子装置

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114786060A (zh) * 2022-02-28 2022-07-22 长沙朗源电子科技有限公司 基于安卓平台无线投屏实现帧率控制的方法及其装置
WO2023169276A1 (zh) * 2022-03-11 2023-09-14 华为技术有限公司 投屏方法、终端设备及计算机可读存储介质
CN115119032A (zh) * 2022-06-22 2022-09-27 北斗星通智联科技有限责任公司 一种可游戏投屏的汽车智能座舱及方法
CN116055623A (zh) * 2022-06-23 2023-05-02 荣耀终端有限公司 一种功耗控制方法及电子设备
CN116055623B (zh) * 2022-06-23 2023-11-03 荣耀终端有限公司 一种功耗控制方法及电子设备、存储介质
CN116033209A (zh) * 2022-08-29 2023-04-28 荣耀终端有限公司 投屏方法和电子设备
CN116033209B (zh) * 2022-08-29 2023-10-20 荣耀终端有限公司 投屏方法和电子设备
CN116737289A (zh) * 2022-09-15 2023-09-12 荣耀终端有限公司 显示方法、终端设备及存储介质

Also Published As

Publication number Publication date
CN113687803A (zh) 2021-11-23

Similar Documents

Publication Publication Date Title
WO2021233218A1 (zh) 投屏方法、投屏源端、投屏目的端、投屏系统及存储介质
CN110109636B (zh) 投屏方法、电子设备以及系统
WO2021175213A1 (zh) 刷新率切换方法和电子设备
WO2022257977A1 (zh) 电子设备的投屏方法和电子设备
WO2021052200A1 (zh) 一种设备能力调度方法及电子设备
CN113691842B (zh) 一种跨设备的内容投射方法及电子设备
CN114040242B (zh) 投屏方法、电子设备和存储介质
WO2022007862A1 (zh) 图像处理方法、系统、电子设备及计算机可读存储介质
CN113923230B (zh) 数据同步方法、电子设备和计算机可读存储介质
CN116360725B (zh) 显示交互系统、显示方法及设备
WO2021190344A1 (zh) 多屏幕显示电子设备和电子设备的多屏幕显示方法
EP4187907A1 (en) Screen projection data processing method and apparatus
WO2021139630A1 (zh) 数据传输方法、装置及数据处理系统、存储介质
CN113986162B (zh) 图层合成方法、设备及计算机可读存储介质
WO2022127632A1 (zh) 一种资源管控方法及设备
WO2021052388A1 (zh) 一种视频通信方法及视频通信装置
US20230350631A1 (en) Projection display method and electronic device
WO2022143310A1 (zh) 一种双路投屏的方法及电子设备
WO2022152174A9 (zh) 一种投屏的方法和电子设备
CN114827696B (zh) 一种跨设备的音视频数据同步播放的方法和电子设备
CN115686403A (zh) 显示参数的调整方法、电子设备、芯片及可读存储介质
CN113747056A (zh) 拍照方法、装置及电子设备
WO2023005900A1 (zh) 一种投屏方法、电子设备及系统
WO2023283941A1 (zh) 一种投屏图像的处理方法和装置
WO2022022405A1 (zh) 屏幕显示的方法、装置、电子设备、计算机存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21809634

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21809634

Country of ref document: EP

Kind code of ref document: A1