CN114938408A - Data transmission method, system, equipment and medium of cloud mobile phone - Google Patents
Data transmission method, system, equipment and medium of cloud mobile phone Download PDFInfo
- Publication number
- CN114938408A CN114938408A CN202210476601.4A CN202210476601A CN114938408A CN 114938408 A CN114938408 A CN 114938408A CN 202210476601 A CN202210476601 A CN 202210476601A CN 114938408 A CN114938408 A CN 114938408A
- Authority
- CN
- China
- Prior art keywords
- data
- screen data
- screen
- mobile phone
- memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 230000005540 biological transmission Effects 0.000 title claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 18
- 230000008859 change Effects 0.000 claims abstract description 12
- 230000004044 response Effects 0.000 claims abstract description 10
- 230000006870 function Effects 0.000 claims description 28
- 230000008569 process Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 5
- 230000007246 mechanism Effects 0.000 claims description 4
- 238000009877 rendering Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 238000013481 data capture Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
- H04M1/72409—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality by interfacing with external accessories
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
- H04L67/303—Terminal profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M1/00—Substation equipment, e.g. for use by subscribers
- H04M1/72—Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
- H04M1/724—User interfaces specially adapted for cordless or mobile telephones
- H04M1/72403—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
- H04M1/7243—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages
- H04M1/72439—User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages for image or video messaging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Telephonic Communication Services (AREA)
- Communication Control (AREA)
Abstract
The invention discloses a data transmission method, a system, equipment and a medium of a cloud mobile phone, wherein the method comprises the following steps: after the cloud mobile phone is connected with a client, a virtual screen and a data buffer area are created, and the changed screen data are intercepted in the data buffer area in response to the change of the screen data of the virtual screen; processing the intercepted screen data and storing the processed screen data into a memory; reading the processed screen data from the memory based on the hardware coding card and coding the read screen data; and transmitting the encoded screen data to the client. By the scheme of the invention, the transmission efficiency of the screen data is improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a data transmission method, a data transmission system, data transmission equipment and a data transmission medium of a cloud mobile phone.
Background
With the rapid development of emerging technologies such as big data and cloud computing, technologies such as cloud mobile phones and cloud games become hot spots in the industry. The cloud mobile phone is a cloud server, and multiple android operating systems are virtualized on a linux (an open source operating system) system through a container technology and other technologies, and each android operating system is equivalent to one cloud mobile phone. A user can realize a scene that a picture is on a terminal and the mobile phone runs on the cloud by connecting a specific cloud mobile phone at a client. The cloud mobile phone displayed on the terminal equipment can enjoy the strong computing power and the storage capacity of the cloud without being limited by the hardware of the terminal equipment. In the implementation process of the cloud mobile phone, because the mobile phone picture is rendered on the cloud, it becomes a key ring to acquire cloud mobile phone rendering data and transmit the cloud mobile phone rendering data to the client through the network.
At present, there are two ways to obtain GPU rendering data inside a cloud mobile phone system, one is to directly read GPU display memory data through an OpenGL (image processing protocol) interface, the other is to directly intercept a synthesized android screen picture, and then send the captured screen data to a MediaCodec (hardware codec, which accelerates decoding and encoding through hardware) interface, the MediaCodec interface can use software coding or call an OpenMax (Open Media access, Open multimedia Acceleration layer) interface to send to a coding card for hardware coding, and then form a video stream to send out through a network.
The current cloud mobile phone video stream is realized with the following defects: the GPU video memory data is directly read through an OpenGL interface, the interface can affect the clock period of a CPU, all tasks can be forced to be completed inside the GPU, a GPU rendering pipeline is stopped, and the efficiency is low. In the aspect of original video stream coding, if the MediaCodec interface calls the OpenMax interface to further call the coding card to perform hardware coding on the original video stream, since a layer of OpenMax is added in the calling process, the system overhead is increased, and further the network delay between the cloud mobile phone and the client is increased.
Disclosure of Invention
In view of this, the invention provides a data transmission method, system, device and medium for a cloud mobile phone, which do not need to use an OpenGL interface to obtain GPU rendering data (i.e., screen data of the cloud mobile phone), so that low efficiency is avoided, and a hardware coding card is directly called to code acquired screen data, so that the coding efficiency is improved, and meanwhile, network delay between the cloud mobile phone and a client is reduced.
Based on the above purpose, an aspect of the embodiments of the present invention provides a data transmission method for a cloud mobile phone, which specifically includes the following steps:
after the cloud mobile phone is connected with a client, a virtual screen and a data buffer area are created, and the changed screen data are intercepted in the data buffer area in response to the change of the screen data of the virtual screen;
processing the intercepted screen data and storing the processed screen data to an internal memory;
reading the processed screen data from the memory based on the hardware coding card and coding the read screen data;
and transmitting the encoded screen data to the client.
In some embodiments, after the step of intercepting the changed screen data in the data buffer in response to a change in the screen data of the virtual screen, the method further comprises:
setting a callback function, monitoring the intercepted screen data in the data buffer area based on the callback function, responding to the generation of newly intercepted screen data, and executing the callback function to finish intercepting the changed screen data.
In some embodiments, processing the captured screen data and storing the processed screen data in a memory includes:
acquiring a latest Image object;
acquiring a pixel matrix of newly intercepted screen data from the Image object;
acquiring a corresponding byte buffer area from the pixel matrix, wherein the byte buffer area contains newly intercepted screen data in a preset data format;
and storing the corresponding byte buffer area into a memory so as to store the newly intercepted screen data with the preset data format into the memory.
In some embodiments, encoding the processed screen data includes:
the device name of the hardware coding card is transparently transmitted to the cloud mobile phone;
compiling a software development kit of a hardware coding card and android source codes of a cloud mobile phone into a shared library, and adapting the shared library to the cloud mobile phone;
and calling an API (application programming interface) of the shared library to read the processed screen data in the memory for coding.
In some embodiments, before the step of creating a virtual screen and a data buffer after the cloud mobile phone establishes the connection with the client, the method further includes:
acquiring screen metadata information of a cloud mobile phone;
and establishing connection with the client based on the screen metadata information.
In some embodiments, creating the virtual screen comprises:
the method comprises the steps of obtaining SurfaceControl based on an android reflection mechanism, and creating a virtual screen based on a createDisplay interface in the SurfaceControl to copy an original screen of the cloud mobile phone.
Creating the data buffer includes:
creating an ImageReader class based on the resolution information of the screen and the preset data format of the screen data;
and calling a getSurface interface of the ImageReader class to create a Surface data buffer area.
In some embodiments, the preset data format is an RGB format;
after the step of storing the corresponding byte buffer into the memory to store the newly captured screen data in the preset data format into the memory, the method further includes:
and reading the newly captured screen data in the RGB format from the memory, converting the screen data in the RGB format into screen data in the YUV format, and storing the screen data in the memory again.
In another aspect of the embodiments of the present invention, there is also provided a video acquisition system for a cloud mobile phone, including:
the intercepting module is configured to create a virtual screen and a data buffer area after the cloud mobile phone is connected with a client, respond to the change of screen data of the virtual screen, and intercept the changed screen data in the data buffer area;
the processing module is configured to process the intercepted screen data and store the processed screen data to the memory;
the coding module is configured to read the processed screen data from the memory based on the hardware coding card and code the read screen data;
a transmission module configured to transmit the encoded screen data to the client.
In another aspect of the embodiments of the present invention, there is also provided a computer device, including: at least one processor; and a memory storing a computer program executable on the processor, the computer program when executed by the processor implementing the steps of the method as above.
In another aspect of the embodiments of the present invention, a computer-readable storage medium is further provided, in which a computer program for implementing the above method steps is stored when the computer program is executed by a processor.
The invention has at least the following beneficial technical effects: by creating a virtual screen and a data buffer area to intercept screen data, the screen data interception efficiency is improved, and further the video transmission efficiency is improved; and the hardware coding card is directly called to code the acquired screen data, so that the screen data coding efficiency is improved, and the network delay between the cloud mobile phone and the client is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by using the drawings without creative efforts.
Fig. 1 is a block diagram of an embodiment of a data transmission method of a cloud mobile phone according to the present invention;
fig. 2 is a schematic diagram of an embodiment of a video capture system of a cloud mobile phone according to the present invention;
FIG. 3 is a schematic structural diagram of an embodiment of a computer device provided in the present invention;
fig. 4 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are only used for convenience of expression and should not be construed as a limitation to the embodiments of the present invention, and no description is given in the following embodiments.
Based on the above purpose, a first aspect of the embodiments of the present invention provides an embodiment of a data transmission method for a cloud mobile phone. As shown in fig. 1, it includes the following steps:
s10, after the cloud mobile phone is connected with the client, a virtual screen and a data buffer area are created, the changed screen data are intercepted in the data buffer area in response to the change of the screen data of the virtual screen;
s20, processing the captured screen data and storing the processed screen data into a memory;
s30, reading the processed screen data from the memory based on the hardware coding card and coding the read screen data;
and S40, transmitting the encoded screen data to the client.
Specifically, after a cloud mobile phone is connected with a client, wherein the client refers to the mobile phone, a virtual display is created as a virtual screen by acquiring a surfacontrol interface of an android system (namely, the cloud mobile phone) and a createDisplay interface of the surfacontrol interface, and original screen data of the cloud mobile phone is copied in real time through the virtual screen; creating a surface as a data buffer area by calling an ImageReader class of an android, intercepting changed screen data when a screen of a virtual screen is changed, calling and executing a callback function when the screen data is changed, and processing the intercepted screen data in the callback function; and directly transmitting the coding card into the container, compiling a software development kit of the coding card along with an android system, calling an API (application programming interface) of the coding card development kit, transmitting the processed screen data to a hardware coding card for coding, and finally transmitting the coded data to a user through a network.
In the embodiment, the screen data of the cloud mobile phone is not required to be acquired by using an OpenGL interface, so that the screen data interception efficiency is improved, and the video transmission efficiency is further improved; and the hardware coding card is directly called to code the acquired screen data, so that the screen data coding efficiency is improved, and the network delay between the cloud mobile phone and the client is reduced.
In some embodiments, after the step of intercepting the changed screen data in the data buffer in response to a change in the screen data of the virtual screen, the method further comprises:
setting a callback function, monitoring the intercepted screen data in the data buffer area based on the callback function, responding to the generation of newly intercepted screen data, and executing the callback function to finish intercepting the changed screen data.
Specifically, a VirtualDisplay is created. VirtualDisplay is a virtual screen provided by the android system, which is a copy of the original screen, which exists in memory, for "producer" as screen data is generated.
And constructing a Surface data buffer area. The Surface is a data buffer provided by a screen data consumer to a producer of the screen data, and the producer (i.e. virtual display) can produce image content on the Surface, and the consumer can consume the produced data on the screen (draw out) or convert the produced data into data desired by the consumer. And allowing an application program of the cloud mobile phone to directly access image data of the Surface by using the Surface created by the Imagereader class of the android system.
And calling a setOnImageAvailableListener interface in the ImageReader class to set a callback function, monitoring newly intercepted screen data on the Surface based on the callback function, and executing the callback function to finish intercepting the new screen data when the new screen data appears so as to enter the next step of processing the screen data.
In this embodiment, an original screen of a cloud mobile phone is copied by creating VirtualDisplay; constructing a Surface data buffer area to enable screen change of the VirtualDisplay to be produced on the Surface; the callback function is set, the screen data in the Surface are monitored based on the callback function, the response is generated by the new screen data, when the new screen capture data appear, the callback function is executed to complete the capture of the changed screen data, the whole process does not need to use an OpenGL interface to obtain the screen data of the cloud mobile phone, the screen data capture efficiency is improved, and the video transmission efficiency is further improved.
In some embodiments, processing the captured screen data and storing the processed screen data in a memory includes:
acquiring a latest Image object;
acquiring a pixel matrix of newly intercepted screen data from the Image object;
acquiring a corresponding byte buffer area from the pixel matrix, wherein the byte buffer area contains newly intercepted screen data in a preset data format;
and storing the corresponding byte buffer area into a memory so as to store the newly intercepted screen data with the preset data format into the memory.
Specifically, in the callback function, an acquireLatestImage interface in the ImageReader class is called to obtain a latest Image object, and screen data information is contained in the Image object; a pixel matrix Plane of the latest Image data (i.e., screen data) in the Image object is acquired. Here, Plane is a class; calling a getBuffer interface in the Plane to obtain a byte buffer area ByteBuffer containing the latest generated screen data in the Plane, wherein the ByteBuffer contains a frame of preset screen data in a data format, such as screen data in an RGB format; and storing the acquired byte buffer into a memory. The newly generated screen data is thus acquired. The hardware coding card can be directly called by the screen data stored in the memory through the scheme, the screen data in the memory is read for coding, the screen data coding efficiency is improved, and the network delay between the cloud mobile phone and the client is reduced.
In some embodiments, encoding the processed screen data includes:
the device name of the hardware coding card is transparently transmitted to the cloud mobile phone;
compiling a software development kit of a hardware coding card and android source codes of a cloud mobile phone into a shared library, and adapting the shared library to the cloud mobile phone;
and calling an API (application programming interface) of the shared library to read the processed screen data in the memory for coding.
Specifically, the hardware coding card device is firstly transmitted to the cloud mobile phone. If the cloud mobile phone wants to directly use the hardware coding card for coding the screen data in the android container of the host, the cloud mobile phone needs to directly expose the device name of the hardware coding card on the host in the android container, for example, the device name of the hardware coding card is exposed in the android container by using a docker command option-device parameter, so that an application program of the cloud mobile phone in the android container can access the hardware coding card.
And then compiling the software development kit of the hardware coding card and the android source code of the cloud mobile phone together to form a shared library, and adapting to an android system in the form of the shared library so that the application program of the cloud mobile phone can access the hardware coding card based on the shared library.
And finally calling an API in the shared library, reading screen data in a preset data format stored in the memory, and encoding, wherein the format output after encoding can be a format supported by H264 or other encoding cards.
According to the embodiment, the hardware coding card is directly called, the screen data in the memory is read for coding, the screen data coding efficiency is improved, and the network delay between the cloud mobile phone and the client is reduced.
In some embodiments, before the step of creating a virtual screen and a data buffer after the cloud mobile phone establishes the connection with the client, the method further includes:
acquiring screen metadata information of a cloud mobile phone;
and establishing connection with the client based on the screen metadata information.
Specifically, before collecting the cloud mobile phone screen data, metadata information of the cloud mobile phone screen needs to be obtained first. The metadata information is some parameter information of the specified cloud mobile phone when the cloud mobile phone is created, and on one hand, the metadata information of the cloud mobile phone is sent to the client as a part of a transmission protocol, and on the other hand, the metadata information is used as a later called parameter. The metadata information includes a version number (occupying 1 byte) of the cloud mobile phone video stream acquisition software, a cloud mobile phone screen resolution (width and height respectively occupying 4 bytes and totaling 8 bytes), a cloud mobile phone screen refresh rate FPS (occupying 1 byte), and a rotation angle (1 byte) of the cloud mobile phone screen. The metadata information of the cloud mobile phone occupies 11 bytes, 8 bits of 1(1 byte) are added after the metadata information, the information of the 1 byte is used as a mark for ending the transmission protocol and is used for checking the integrity of a protocol header when a client receives the metadata information, when the last byte of the check protocol is continuous 8 bits of 1, the protocol header is ended, and at the moment, the client can receive the picture data. The metadata information is 12 bytes in total. The 12 bytes of information is used as header information of a cloud mobile phone and a client transmission protocol. When the cloud mobile phone video stream acquisition process runs, 11 bytes of metadata information of a cloud mobile phone screen is firstly acquired, when the cloud mobile phone is connected with a client, the 11 bytes and 1 byte are firstly added to be used as the metadata information of header information of a transmission protocol and sent to the client, the client analyzes and checks the received information, and if a normal protocol header is found, the client is ready to receive the following data stream. It should be noted that the method provided by the present invention can support 720p and 1080p resolutions, but is not limited thereto, wherein the screen size corresponding to 720p is 1280 in width, and the screen sizes corresponding to 720p and 1080p are 1920 in width and 1080 in height. FPS supports 30 and 60, but is not so limited. The screen direction of the cloud mobile phone supports a horizontal screen and a vertical screen, and the horizontal screen and the vertical screen are replaced by 0 and 90 in metadata information respectively.
When the client is connected for the first time, the cloud mobile phone screen information obtained and configured in the step S1 needs to be sent to the client, and the information needs to be sent only once in the process of cloud mobile phone connection. And when the cloud mobile phone is disconnected from the client and is connected again, the cloud mobile phone is sent again. The picture data can be transmitted only after the protocol header is transmitted.
In some embodiments, creating the virtual screen comprises:
the method comprises the steps of obtaining SurfaceControl based on an android reflection mechanism, and creating a virtual screen based on a createDisplay interface in the SurfaceControl to copy an original screen of the cloud mobile phone.
Creating the data buffer includes:
creating an ImageReader class based on the resolution information of the screen and the preset data format of the screen data;
and calling a getSurface interface of the ImageReader class to create a Surface data buffer area.
Specifically, a VirtualDisplay is created. VirtualDisplay is a virtual screen provided by the android system, which is a copy of the original screen, which exists in memory for the "producer" to be generated as screen data. The android system provides mediaproject manager service, and creates virtual display according to the constructed Surface data buffer, but in the scheme, context is needed to obtain mediaproject manager, an authorized window pops up on a mobile phone screen, and screen capture can be performed only after a user agrees (namely, the method is mentioned in the invention).
Therefore, a higher-authority screen capture process is required, and a SurfaceControl is acquired by using a reflection mechanism of the android, wherein the SurfaceControl is an API hidden in the android system; the createDisplay interface of SurfaceControl is then utilized to create virtualdisplayment.
And constructing a Surface data buffer area. The Surface is a data buffer provided by a screen data consumer to a producer of the screen data, and the producers (i.e. virtual display) can produce image content on the Surface, and the consumers can consume the produced data on the screen (draw out) or convert the produced data into data desired by the consumers. The specific process for constructing the Surface is as follows:
firstly, acquiring resolution information of a screen and data format information of an object meeting the video stream data format requirement, and creating an ImageReader class based on the resolution information and the data format information of the screen, for example, the data format of the object can be set to be RGB); and secondly, calling a getSurface interface of the ImageReader class to acquire the Surface. And allowing an application program of the cloud mobile phone to directly access image data of the Surface by using the Surface created by the Imagereader class of the android system.
And calling a setOnImageAvailableListener interface in the ImageReader class to set a callback function, monitoring screen data produced by the VirtualDisplay on the Surface based on the callback function, and executing the callback function when new screen data appears so as to enter the next screen data processing.
In this embodiment, an original screen of a cloud mobile phone is copied by creating VirtualDisplay; constructing a Surface data buffer area so that the screen change of the virtual display is produced on the Surface; the callback function is set, the screen data in the Surface are monitored based on the callback function, the response is generated by the new screen data, when the new screen capture data appear, the callback function is executed to complete the capture of the changed screen data, the whole process does not need to use an OpenGL interface to obtain the screen data of the cloud mobile phone, the screen data capture efficiency is improved, and the video transmission efficiency is further improved.
In some embodiments, the preset data format is an RGB format;
after the step of storing the corresponding byte buffer into a memory to store the newly intercepted screen data in the preset data format into the memory, the method further includes:
and reading newly intercepted screen data in the RGB format from the memory, converting the screen data in the RGB format into screen data in the YUV format, and storing the screen data in the memory again.
Specifically, the YUV format includes YUV420, YUV422, YUV444, and the like, but is not limited thereto. The YUV data format occupies a small storage space.
The screen data in the RGB format is converted into the screen data in the YUV format and then stored in the memory again to reduce the occupation of the memory space, so that the hardware coding card saves the reading time when reading the screen data from the memory and improves the coding efficiency.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 2, an embodiment of the present invention further provides a video capture system of a cloud mobile phone, which includes:
an intercepting module 110, wherein the intercepting module 110 is configured to create a virtual screen and a data buffer area after the cloud mobile phone establishes a connection with a client, and intercept the changed screen data in the data buffer area in response to a change of screen data of the virtual screen;
the processing module 120, the processing module 120 is configured to process the intercepted screen data and store the processed screen data to the memory;
the encoding module 130 is configured to read the processed screen data from the memory based on a hardware encoding card and encode the read screen data;
a transmission module 140, the transmission module 140 configured to transmit the encoded screen data to the client.
Specifically, after the cloud mobile phone is connected with the client, the intercepting module acquires a SurfaceControl interface of an android system (namely, the cloud mobile phone), creates a virtual display as a virtual screen through a createDisplay interface of the SurfaceControl interface, and copies original screen data of the cloud mobile phone in real time through the virtual screen; the intercepting module is also used for creating a surface serving as a data buffer area by calling an Imagereader class of an android and is used for intercepting the changed screen data when the screen of the virtual screen is changed, and calling and executing a callback function when the screen data is changed; the processing module processes the intercepted screen data in the callback function; the coding module directly transmits the coding card to the inside of the container, then the software development kit of the coding card is compiled along with an android system, then an API of the coding card development kit is called, the processed screen data is sent to the hardware coding card for coding, and finally the transmission module transmits the coded data to a user through a network.
In the embodiment, the screen data of the cloud mobile phone is not required to be acquired by using an OpenGL interface, so that the screen data interception efficiency is improved, and the video transmission efficiency is further improved; and the hardware coding card is directly called to code the acquired screen data, so that the screen data coding efficiency is improved, and the network delay between the cloud mobile phone and the client is reduced.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 3, the embodiment of the present invention further provides a computer device 30, in which the computer device 30 comprises a processor 310 and a memory 320, the memory 320 stores a computer program 321 that can run on the processor, and the processor 310 executes the program to perform the steps of the above method.
The memory is used as a nonvolatile computer readable storage medium, and may be used to store a nonvolatile software program, a nonvolatile computer executable program, and modules, such as program instructions/modules corresponding to the data transmission method of the cloud mobile phone in this embodiment of the present application. The processor executes various functional applications and data processing of the device by running the nonvolatile software program, instructions and modules stored in the memory, that is, the data transmission method of the cloud mobile phone of the above method embodiment is realized.
The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the device, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and such remote memory may be coupled to the local module via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
Based on the same inventive concept, according to another aspect of the present invention, as shown in fig. 4, an embodiment of the present invention further provides a computer-readable storage medium 40, where the computer-readable storage medium 40 stores a computer program 410, which when executed by a processor, performs the above method.
Finally, it should be noted that, as will be understood by those skilled in the art, all or part of the processes of the methods of the above embodiments may be implemented by a computer program, which may be stored in a computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. The storage medium of the program may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like. The embodiments of the computer program may achieve the same or similar effects as any of the above-described method embodiments.
Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as software or hardware depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments of the present invention.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. The numbers of the embodiments disclosed in the above embodiments of the present invention are merely for description, and do not represent the advantages or disadvantages of the embodiments. Furthermore, although elements of the disclosed embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of the embodiments of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements and the like that may be made without departing from the spirit or scope of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (10)
1. A data transmission method of a cloud mobile phone is characterized by comprising the following steps:
after the cloud mobile phone is connected with a client, a virtual screen and a data buffer area are created, and the changed screen data are intercepted in the data buffer area in response to the change of the screen data of the virtual screen;
processing the intercepted screen data and storing the processed screen data to an internal memory;
reading the processed screen data from the memory based on the hardware coding card and coding the read screen data;
and transmitting the encoded screen data to the client.
2. The method of claim 1, further comprising, after the step of intercepting, in response to a change in screen data of the virtual screen, the changed screen data in the data buffer:
setting a callback function, monitoring the intercepted screen data in the data buffer area based on the callback function, responding to the generation of newly intercepted screen data, and executing the callback function to finish intercepting the changed screen data.
3. The method of claim 1, wherein processing the captured screen data and saving the processed screen data to a memory comprises:
acquiring a latest Image object;
acquiring a pixel matrix of newly intercepted screen data from the Image object;
acquiring a corresponding byte buffer area from the pixel matrix, wherein the byte buffer area contains newly intercepted screen data in a preset data format;
and storing the corresponding byte buffer area into a memory so as to store the newly intercepted screen data with the preset data format into the memory.
4. The method of claim 1, wherein encoding the processed screen data comprises:
the device name of the hardware coding card is transparently transmitted to the cloud mobile phone;
compiling a software development kit of a hardware coding card and android source codes of a cloud mobile phone into a shared library, and adapting the shared library to the cloud mobile phone;
and calling the API of the shared library to read the processed screen data in the memory for coding.
5. The method according to claim 1, wherein before the step of creating the virtual screen and the data buffer after the cloud mobile phone establishes the connection with the client, the method further comprises:
acquiring screen metadata information of a cloud mobile phone;
and establishing connection with the client based on the screen metadata information.
6. The method of claim 1, wherein creating a virtual screen comprises:
acquiring SurfaceControl based on an android reflection mechanism, and creating a virtual screen based on a createDisplay interface in the SurfaceControl to copy an original screen of a cloud mobile phone;
creating the data buffer includes:
creating an ImageReader class based on the resolution information of the screen and the preset data format of the screen data;
and calling a getSurface interface of the ImageReader class to create a Surface data buffer area.
7. The method of claim 3, wherein the predetermined data format is an RGB format;
after the step of storing the corresponding byte buffer into a memory to store the newly intercepted screen data in the preset data format into the memory, the method further includes:
and reading newly intercepted screen data in the RGB format from the memory, converting the screen data in the RGB format into screen data in the YUV format, and storing the screen data in the memory again.
8. A video acquisition system of a cloud mobile phone is characterized by comprising:
the system comprises an intercepting module, a data buffer area and a display module, wherein the intercepting module is configured to create a virtual screen and the data buffer area after the cloud mobile phone is connected with a client, respond to the change of screen data of the virtual screen, and intercept the changed screen data in the data buffer area;
the processing module is configured to process the intercepted screen data and store the processed screen data to the memory;
the coding module is configured to read the processed screen data from the memory based on the hardware coding card and code the read screen data;
a transmission module configured to transmit the encoded screen data to the client.
9. A computer device, comprising:
at least one processor; and
memory storing a computer program operable on the processor, characterized in that the processor executes the program to perform the steps of the method according to any of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210476601.4A CN114938408B (en) | 2022-04-30 | 2022-04-30 | Data transmission method, system, equipment and medium of cloud mobile phone |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210476601.4A CN114938408B (en) | 2022-04-30 | 2022-04-30 | Data transmission method, system, equipment and medium of cloud mobile phone |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114938408A true CN114938408A (en) | 2022-08-23 |
| CN114938408B CN114938408B (en) | 2023-07-14 |
Family
ID=82863726
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210476601.4A Active CN114938408B (en) | 2022-04-30 | 2022-04-30 | Data transmission method, system, equipment and medium of cloud mobile phone |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114938408B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116828243A (en) * | 2023-06-05 | 2023-09-29 | 启朔(深圳)科技有限公司 | Hardware encoding and decoding method, mobile terminal, computer device and storage medium |
| CN118175157A (en) * | 2024-05-09 | 2024-06-11 | 江苏北弓智能科技有限公司 | Remote mobile cloud desktop acquisition method |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104660687A (en) * | 2015-02-02 | 2015-05-27 | 上海视聪网络信息技术有限公司 | Realization method and system for virtual desktop display |
| CN105988688A (en) * | 2016-01-22 | 2016-10-05 | 腾讯科技（深圳）有限公司 | Screen capture method and apparatus |
| CN110166792A (en) * | 2018-02-14 | 2019-08-23 | 阿里巴巴集团控股有限公司 | Image display method, device, system and storage medium |
| CN112003881A (en) * | 2020-10-28 | 2020-11-27 | 湖南天琛信息科技有限公司 | Safety cloud mobile phone system based on private cloud |
-
2022
- 2022-04-30 CN CN202210476601.4A patent/CN114938408B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104660687A (en) * | 2015-02-02 | 2015-05-27 | 上海视聪网络信息技术有限公司 | Realization method and system for virtual desktop display |
| CN105988688A (en) * | 2016-01-22 | 2016-10-05 | 腾讯科技（深圳）有限公司 | Screen capture method and apparatus |
| CN110166792A (en) * | 2018-02-14 | 2019-08-23 | 阿里巴巴集团控股有限公司 | Image display method, device, system and storage medium |
| CN112003881A (en) * | 2020-10-28 | 2020-11-27 | 湖南天琛信息科技有限公司 | Safety cloud mobile phone system based on private cloud |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116828243A (en) * | 2023-06-05 | 2023-09-29 | 启朔(深圳)科技有限公司 | Hardware encoding and decoding method, mobile terminal, computer device and storage medium |
| CN118175157A (en) * | 2024-05-09 | 2024-06-11 | 江苏北弓智能科技有限公司 | Remote mobile cloud desktop acquisition method |
| CN118175157B (en) * | 2024-05-09 | 2024-08-02 | 江苏北弓智能科技有限公司 | Remote mobile cloud desktop acquisition method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114938408B (en) | 2023-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12003743B2 (en) | Video stream decoding method and apparatus, terminal device, and storage medium | |
| CN111681167B (en) | Image quality adjusting method and device, storage medium and electronic equipment | |
| US10110936B2 (en) | Web-based live broadcast | |
| WO2022257699A1 (en) | Image picture display method and apparatus, device, storage medium and program product | |
| CN110430441B (en) | Cloud mobile phone video acquisition method, system, device and storage medium | |
| CN114938408B (en) | Data transmission method, system, equipment and medium of cloud mobile phone | |
| CN113457160B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN112843676B (en) | Data processing method, device, terminal, server and storage medium | |
| CN112422868A (en) | Data processing method, terminal device and server | |
| CN110730364B (en) | Cloud mobile phone data transmission method and device and storage medium | |
| CN110049347B (en) | Method, system, terminal and device for configuring images on live interface | |
| CN115089966A (en) | Video rendering method and system applied to cloud game and related equipment | |
| CN113973224B (en) | Media information transmission method, computing device and storage medium | |
| CN113327303B (en) | Image processing method, image processing device, computer equipment and storage medium | |
| KR102238091B1 (en) | System and method for 3d model compression and decompression | |
| CN114245137A (en) | Video frame processing method performed by GPU and video frame processing apparatus including GPU | |
| US20140327698A1 (en) | System and method for hybrid graphics and text rendering and client computer and graphics processing unit incorporating the same | |
| CN115225881A (en) | Data transmission method, device, equipment and storage medium | |
| CN113301290A (en) | Video data processing method and video conference terminal | |
| CN115225615B (en) | Illusion engine pixel streaming method and device | |
| CN112995134B (en) | Three-dimensional video streaming media transmission method and visualization method | |
| CN116170636A (en) | Live video playing method and device, equipment and medium thereof | |
| CN116546262A (en) | Data processing method, device, equipment and storage medium | |
| CN107005731B (en) | Image cloud end streaming media service method, server and system using application codes | |
| CN110798715A (en) | Video playing method and system based on image string |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |