WO2023045551A1 - 摄像头控制方法、系统、电子设备和存储介质 - Google Patents

摄像头控制方法、系统、电子设备和存储介质 Download PDF

Info

Publication number
WO2023045551A1
WO2023045551A1 PCT/CN2022/108302 CN2022108302W WO2023045551A1 WO 2023045551 A1 WO2023045551 A1 WO 2023045551A1 CN 2022108302 W CN2022108302 W CN 2022108302W WO 2023045551 A1 WO2023045551 A1 WO 2023045551A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
server
image frames
cache queue
client
Prior art date
Application number
PCT/CN2022/108302
Other languages
English (en)
French (fr)
Inventor
赵圣阳
陈浩
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2023045551A1 publication Critical patent/WO2023045551A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
    • H04N23/661—Transmitting camera control signals through networks, e.g. control via the Internet
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording

Definitions

  • the embodiments of the present application relate to the technical field of the Internet, and in particular to a method for controlling a camera, an electronic device, and a storage medium.
  • Cloud mobile phone is a mobile phone that applies cloud computing technology to network terminal services and realizes cloud services through cloud servers. In fact, it is a smartphone deeply integrated with network services. This type of mobile phone can realize many functions through the network by virtue of its own system and the network terminal set up by the manufacturer.
  • the operating system of the cloud mobile phone also known as the mobile phone virtual machine
  • the method of camera redirection is generally adopted, that is, the connection between the virtual machine of the mobile phone and the client is established, and the camera of the client is mapped to the virtual machine of the mobile phone.
  • This solution is likely to affect the real-time performance of the image after the camera is redirected, thus affecting the user experience.
  • An embodiment of the present application provides a method for controlling a camera, which is applied to a client, and includes: receiving a camera control instruction sent by a virtual terminal running in a server; controlling the camera to collect image frames according to the control instruction, and The image frame is stored in a cache queue; when the number of image frames in the cache queue is greater than a preset number threshold, the target image frame in the cache queue is cleared; wherein, the time stamp of the target image frame The time interval between the current time stamp and the current time stamp is greater than the preset interval; the remaining image frames in the cache queue are sent to the server so that the server sends the received image frames to the operating system of the virtual terminal
  • the camera hardware abstraction layer within within.
  • the embodiment of the present application also provides a method for controlling a camera, which is applied to a server in which a virtual terminal runs, and the method includes: sending a camera control instruction to a client; A control instruction, controlling the camera to collect image frames, and storing the image frames in a cache queue; when the number of image frames in the cache queue is greater than a preset number threshold, clearing the objects in the cache queue image frame and send the remaining image frames in the cache queue to the server; wherein, the time interval between the time stamp of the target image frame and the current time stamp is greater than a preset time interval; receiving the client The image frame is sent, and the image frame is sent to the camera hardware abstraction layer in the operating system of the virtual terminal.
  • the embodiment of the present application also provides a camera control system, including: a server and a client, wherein a virtual terminal runs in the server; the server is used to send a control command to the camera to the client; the client It is used to control the camera to collect image frames according to the control instruction, and store the image frames in a cache queue; the client is also configured to have the number of image frames in the cache queue greater than a preset number threshold In the case of , clear the target image frame in the cache queue, and send the remaining image frames in the cache queue to the server; wherein, the time between the timestamp of the target image frame and the current timestamp The interval is greater than the preset interval; the server is also used to send the received image frame to the camera hardware abstraction layer in the operating system of the virtual terminal.
  • a camera control system including: a server and a client, wherein a virtual terminal runs in the server; the server is used to send a control command to the camera to the client; the client It is used to control the camera to collect image frames according to the
  • the embodiment of the present application also provides an electronic device, including: at least one processor; and a memory connected in communication with the at least one processor; wherein, the memory stores information that can be executed by the at least one processor. Instructions; when the electronic device is a client, the instructions are executed by the at least one processor, so that the at least one processor can execute the camera control method applied to the client; when the electronic device is a server In the case where, the instructions are executed by the at least one processor, so that the at least one processor can execute the camera control method applied to the server.
  • the embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and is characterized in that, when the computer program is executed by a processor, the camera control method applied to the client is implemented, or the camera control method applied to the server is implemented. method.
  • Fig. 1 is a schematic diagram of the application scenario mentioned in the embodiment of the present application.
  • FIG. 2 is a schematic flowchart of a camera control method applied to a client mentioned in an embodiment of the present application
  • FIG. 3 is a schematic flowchart of a camera control method applied to a server mentioned in the embodiment of the present application;
  • Fig. 4 is a schematic diagram of a camera control system mentioned in the embodiment of the present application.
  • FIG. 5 is a schematic flowchart of another camera control method mentioned in the embodiment of the present application.
  • Fig. 6 is a schematic structural diagram of the electronic device mentioned in the embodiment of the present application.
  • the main purpose of the embodiments of the present application is to provide a camera control method, an electronic device, and a computer-readable storage medium, so that the real-time performance of a picture after camera redirection can be improved, thereby improving user experience.
  • the cloud mobile phone system includes: a client (such as client 1, client 2, and client 3 in FIG. 1 ), and a server.
  • the server may include: a login management module, a virtual machine management module, and a mobile phone virtual machine (such as mobile phone virtual machine 1, mobile phone virtual machine 2, and mobile phone virtual machine 3 in FIG. 1 ) running in the server.
  • the client is used to initiate a login request to the server.
  • the server verifies the login account. After the verification is passed, a mobile virtual machine is assigned to the corresponding login user, and the connection parameters of the mobile virtual machine are returned to the client.
  • the client The connection link with the corresponding mobile phone virtual machine can be established through the returned connection parameters. After the connection is successful, the corresponding mobile phone virtual machine can be operated through the client.
  • the server runs a login management module, a virtual machine management module, etc.
  • the login management module is used to manage user information and login verification;
  • the virtual machine management module is used to manage the virtual machine of the mobile phone, including the creation, deletion, start, and stop of the virtual machine of the mobile phone, and can also be used to manage the running status of the virtual machine of the mobile phone .
  • the mobile phone virtual machine is an independent system running on the server, and the client can establish a connection with it, and operate the mobile phone virtual machine through the established link.
  • some preset modules can also be deployed on the mobile virtual machine, through which the mobile virtual machine can be managed and controlled.
  • the embodiment of the present application provides a method for controlling a camera, which is applied to a client.
  • the client can be a client in the cloud mobile phone system in FIG. 1, and the client can be an ordinary mobile phone device, but when using a cloud mobile phone, the client can only function as a human-computer interaction.
  • the client is provided with a camera (that is, a camera device), and the number of clients can be multiple.
  • the server can connect to client 1, client 2 and client 3 at the same time, and multiple mobile phones can be run simultaneously on the server.
  • the operating systems of the virtual machines are used by Client 1, Client 2, and Client 3 respectively.
  • the schematic flowchart of the camera control method in this embodiment can refer to FIG. 2, including:
  • Step 201 receiving the camera control instruction sent by the virtual terminal running in the server;
  • Step 202 According to the control instruction, control the camera to collect image frames, and store the image frames in a buffer queue;
  • Step 203 When the number of image frames in the cache queue is greater than the preset number threshold, clear the target image frames in the cache queue;
  • Step 204 Send the remaining image frames in the cache queue to the server, so that the server sends the received image frames to the camera hardware abstraction layer in the operating system of the virtual terminal.
  • the client when the number of image frames in the cache queue is greater than the preset number threshold, it means that the current network status is not good and there are a large number of image frames in the cache queue that cannot be reported in time, that is, a large number of image frames are accumulated in the cache queue At this time, the client will clear the target image frame in the cache queue.
  • the time interval between the timestamp of the target image frame and the current timestamp is greater than the preset interval, which means that the target image frame was stored in the cache queue a long time ago.
  • the image frames in can be regarded as expired data, that is, in the embodiment of the present application, when a large number of image frames are accumulated in the cache queue, the expired data in the cache queue can be cleared in time.
  • the client sends the remaining image frames in the cache queue to the server so that the server sends the received image frames to the camera hardware abstraction layer in the operating system of the virtual terminal. That is to say, the client can send newer image frames to the server in time, so that the server sends these newer image frames to the camera hardware abstraction layer in the operating system of the virtual terminal, avoiding a large number of image frames in the cache queue
  • the accumulation causes a large delay in the image screen after the camera is redirected, which is conducive to improving the real-time performance of the image after the camera is redirected, thereby improving the user experience.
  • the client may receive a camera control instruction sent by a virtual terminal running on the server.
  • the virtual terminal running in the server may be the mobile phone virtual machine in FIG. 1 .
  • the camera hardware abstraction layer in the operating system of the mobile virtual machine can intercept the use request of the APP running in the mobile virtual machine to the camera (also can be understood as the use request of the camera), the use request can carry the use parameters, the use parameters It can include the camera parameters that the APP expects when using the camera, for example, it can include: camera resolution, whether the flash is on, shooting mode (such as portrait mode, landscape mode, night scene mode, etc.), the storage address of the captured image, and turning on the front camera Or the rear camera, frame rate, etc.
  • the camera hardware abstraction layer can generate control instructions for the camera according to the use request carrying the use parameters, and send the camera control instructions to the client.
  • a transmission control protocol Transmission Control Protocol, TCP
  • TCP Transmission Control Protocol
  • the client may control the camera to collect image frames according to the control instruction, and store the image frames in a cache queue.
  • the client can obtain the camera usage parameters of the APP running in the virtual machine of the mobile phone according to the control instructions, and set the camera device in the client, that is, the camera according to the usage parameters, such as setting the camera resolution, whether the flash is on when taking pictures, and the shooting mode (such as portrait mode, landscape mode, night scene mode, etc.), the storage address of the captured image, whether to turn on the front camera or the rear camera, etc.
  • image capture can be performed, that is, image frames can be collected.
  • the client can store the collected image frames in the preset cache queue, and the image frames stored in the cache queue can all have timestamps.
  • the timestamp of an image frame can be understood as the time point when the image frame was collected. It can also be understood as the time point when the captured image frame is stored in the cache queue.
  • step 203 the time interval between the time stamp of the target image frame and the current time stamp is greater than a preset interval.
  • the client can judge whether the number of image frames stored in the cache queue is greater than the preset number threshold, and determine whether the number of image frames stored in the cache queue is greater than the preset number In the case of the threshold, clear the target image frame in the cache queue.
  • the preset number threshold can be set according to actual needs, and is used to indicate that the number of image frames stored in the current buffer queue is too large.
  • this embodiment clears the target image frame in the cache queue, that is, clears the older data in the cache queue, so as to avoid data accumulation in the cache queue, so that the image frame arranged at the head of the cache queue is relatively new data.
  • the camera hardware abstraction layer (Hardware Abstraction Layer, HAL) is located at the interface layer between the operating system kernel and the hardware circuit.
  • HAL Hardware Abstraction Layer
  • the client sends the remaining image frames in the cache queue to the server, so that the server sends the received image frames to the camera hardware abstraction layer in the operating system of the virtual terminal.
  • the client may sequentially send the remaining image frames to the server according to the order in which the remaining image frames enter the cache queue.
  • the client may send the remaining image frames in the cache queue to the server through the TCP link established with the server.
  • the server can send the received image frame to the camera hardware abstraction layer in the operating system of the virtual terminal that initiates the control command to the camera, and then the APP running in the virtual terminal that requests to use the camera can display the camera picture normally.
  • after clearing the target image frame in the cache queue further includes: reducing the frame rate of the camera when there are image frames in the cache queue that have not been sent to the server for more than a preset time period. That is to say, after clearing the target image frame in the cache queue, if it is detected that there are still image frames in the cache queue that have not been sent to the server for a preset period of time, it means that the poor network status does not occur occasionally or for a short time However, it is an inevitable objective situation in the current period of time. At this time, the frame rate of the camera can be reduced, that is, the number of image frames displayed per second can be reduced to adapt to the current objective situation of poor network and ensure that users have a better experience. use experience.
  • reducing the frame rate of the camera includes: determining the sending speed of the image frames in the cache queue to the server; determining the frame rate adjustment amount according to the sending speed, and determining the frame rate adjustment amount according to the frame rate The amount to adjust reduces the frame rate of the camera.
  • the faster the speed of sending image frames to the server it means that the current network speed is relatively fast and the network status is relatively good
  • the slower the speed of sending image frames to the server it means that the current network speed is relatively slow and the network status is relatively good. Difference.
  • the lower the speed of sending image frames to the server, the larger the frame rate adjustment, the larger the speed of sending image frames to the server, the smaller the frame rate adjustment, and the reduced frame rate of the camera is the current frame rate of the camera minus the frame rate rate adjustment. That is to say, the lower the speed of sending image frames to the server, the lower the reduced frame rate, and the higher the speed of sending image frames to the server, the higher the reduced frame rate.
  • the client may pre-store the correspondence between the sending speed and the frame rate adjustment, so that the client may determine the frame rate adjustment according to the pre-stored correspondence and the sending speed of the image frame to the server. This frame rate adjustment method can better adapt to different network states.
  • reducing the frame rate of the camera can also be achieved in the following manner; reduce the frame rate of the camera with a preset step value; wherein, the step value can be set according to actual needs, that is, each time the camera The frame rate is reduced by the size of the step value.
  • the frame rate when reducing the frame rate of the camera, the frame rate is reduced to a preset frame rate at least. That is to say, the frame rate will not be reduced indefinitely, and when the frame rate is the preset frame rate, the frame rate will not be further reduced.
  • the preset frame rate it is possible to reduce the frame rate to adapt to the objective situation of the current poor network while controlling the size of the frame rate reduction, so that the reduced frame rate will not be too small, so that the screen display can be guaranteed to a certain extent of continuity.
  • the preset frame rate can be set according to actual needs, such as setting according to the user's requirements for coherence. If the requirement for picture continuity is low, the preset frame rate can be set relatively small. In a specific implementation, the preset frame rate can be set to 20fps, that is, the frame rate is reduced to 20fps at the lowest.
  • sending the remaining image frames in the cache queue to the virtual terminal running in the server includes: performing H264 encoding on the remaining image frames in the cache queue , to obtain the encoded image frame; send the encoded image frame to the virtual terminal running in the server, so that the virtual terminal performs H264 decoding on the encoded image frame to obtain the decoded image frame.
  • H264 encoding of the image frame a large amount of redundant information in the image frame data can be reduced, and the amount of data to be sent can be reduced, thereby speeding up the speed at which the client sends the image frame to the server, which is beneficial to improve the real-time performance of the camera image to a certain extent. sex.
  • the coding can be soft coding or hard coding.
  • the soft coding is completed by the operation of the central processing unit (Central Processing Unit, CPU), the hard coding is completed by the operation of the graphics processing unit (Graphics Processing Unit, GPU), and the hard coding is completed by the operation of the graphics processing unit (GPU).
  • CPU Central Processing Unit
  • GPU Graphics Processing Unit
  • GPU graphics processing unit
  • the corresponding decoding can be soft decoding or hard decoding.
  • Soft decoding is completed by CPU operation, and hard decoding is completed by GPU operation. The efficiency of hard decoding is much higher than that of soft decoding.
  • the embodiment of the present application also provides a camera control method, which is applied to a server, and a virtual terminal runs in the server.
  • the server may be the server in FIG. include:
  • Step 301 Send a control instruction to the camera to the client;
  • the client controls the camera to collect image frames according to the control instructions, and stores the image frames in the cache queue; when the number of image frames in the cache queue is greater than the preset number threshold, clears the target image frames in the cache queue and Send the remaining image frames in the cache queue to the server; wherein, the time interval between the time stamp of the target image frame and the current time stamp is greater than a preset interval.
  • the server when it detects a request from the virtual terminal to call the camera (that is, a request to use the camera), it may send a camera control instruction to the client according to the request to call the camera.
  • the request of the virtual terminal to call the camera may specifically be a request of an APP running in the virtual terminal to call the camera.
  • Step 302 Receive the image frame sent by the client, and send the image frame to the camera hardware abstraction layer in the operating system of the virtual terminal.
  • receiving the image frame sent by the client includes: receiving the encoded image frame sent by the client; wherein, the client performs H264 encoding on the remaining image frames in the cache queue to obtain the encoded an image frame; performing H264 decoding on the encoded image frame to obtain the decoded image frame; sending the decoded image frame to the camera hardware abstraction layer in the operating system of the virtual terminal.
  • this embodiment is an embodiment applied to a server corresponding to the above embodiment applied to a client, and this embodiment can be implemented in cooperation with the above embodiment applied to a client.
  • the relevant technical details and technical effects mentioned in the above-mentioned embodiment applied to the client are still valid in this embodiment, and will not be repeated here in order to reduce repetition.
  • the relevant technical details mentioned in this embodiment can also be applied to the above-mentioned embodiment applied to the client.
  • step division of the above various methods is only for the sake of clarity of description. During implementation, it can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. ; Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but not changing the core design of the algorithm and process are all within the scope of protection of this patent.
  • the embodiment of the present application also provides a camera control system, including: a server and a client, wherein a virtual terminal runs in the server; the server is used to send a control command to the camera to the client; the client It is used to control the camera to collect image frames according to the control instruction, and store the image frames in a cache queue; the client is also configured to have the number of image frames in the cache queue greater than a preset number threshold In the case of , clear the target image frame in the cache queue, and send the remaining image frames in the cache queue to the server; wherein, the time between the timestamp of the target image frame and the current timestamp The interval is greater than the preset interval; the server is also used to send the received image frame to the camera hardware abstraction layer in the operating system of the virtual terminal.
  • a camera control system including: a server and a client, wherein a virtual terminal runs in the server; the server is used to send a control command to the camera to the client; the client It is used to control the camera to collect image frames according to the
  • the schematic diagram of the camera control system can refer to Fig. 4, including: server 401 and client 402, the virtual terminal running in server 401 includes: camera hardware abstraction layer 4010, H264 decoding module 4011, interface machine Server4012; 402 includes: a camera device 4020 , a frame buffer module 4021 , a camera control module 4022 , an H264 encoding module 4023 , and an interface machine Client4024.
  • the camera hardware abstraction layer 4010 is used to intercept and obtain the use request of the APP running on the virtual terminal (such as in the mobile phone virtual machine) to the camera.
  • the use request carries use parameters, and the camera hardware abstraction layer 4010 converts the use request into controlling the camera
  • the command (also can be referred to as the control command to the camera) is sent to the client 402 through the interface machine Server4012 running in the virtual machine of the mobile phone.
  • the server 401 detects that there is an APP request for using the camera, it can send a notification to the mobile phone virtual machine through the existing link between the client 402 and the mobile phone virtual machine, and then the interface machine Client4024 establishes an interface with the mobile phone virtual machine.
  • the interface machine Server4012 is used to issue commands to control the camera and receive image frames sent by the client.
  • the H264 decoding module 4011 is used to decode the coded image frame received by the interface machine Server4012, and inject the decoded data into the camera hardware abstraction layer 4010.
  • the interface machine Client4024 is used to establish a TCP link with the interface machine Server4012, receive the camera control instruction sent by the interface machine Server4012, and send the H264 encoded image frame to the server.
  • the camera device 4020 is configured to complete camera setting and image capture according to the instruction of controlling the camera, and store the captured image frames into the buffer queue of the frame buffer module 4021 .
  • the camera device may be understood as a camera.
  • the frame buffer module 4021 is used to manage and cache image frames.
  • the number of image frames in the buffer queue is large (that is, the number of image frames stored in the above-mentioned judgment buffer queue is greater than the preset number threshold), then The older data in the cache queue (that is, the above-mentioned target image frame) is discarded to ensure the real-time performance of the camera video.
  • the camera control module 4022 is configured to manage the camera device 4020 and the frame buffer module 4021 in real time according to the current network conditions. In the case of a poor network, a large amount of data will accumulate in the cache queue, causing the server to fail to receive the latest frame data in time and there will be a delay in the picture.
  • the camera control module 4022 detects the amount of data in the cache queue. When the number of image frames is large, the older data (that is, expired data) in the cache queue is cleared in time. When the image frames in the cache queue cannot be sent in time for a long time, the frame rate of the camera will be reduced to avoid a large accumulation of image frames and ensure the real-time performance of the image after the camera is redirected.
  • the H264 encoding module 4023 is used to obtain image frames from the frame buffer module 4021, perform H264 encoding, and send the encoded data to the server through the interface machine Client4024.
  • the camera control method can refer to Figure 5, including:
  • Step 501 The camera hardware abstraction layer in the mobile phone virtual machine detects the APP's request for using the camera, and notifies the client to establish a camera redirection data link with the mobile phone virtual machine running in the server;
  • an interception function for the camera is detected, a notification is sent to the client through the connection link between the client and the mobile virtual machine, and the interface machine Client establishes a connection with the interface machine Server Data link between camera redirection.
  • Step 502 The camera hardware abstraction layer in the mobile phone virtual machine generates an instruction to control the camera according to the APP's request for using the camera, and sends a camera start command to the client through the established link;
  • the instruction for controlling the camera carries parameters for using the camera.
  • the camera hardware abstraction layer can also obtain the parameters of the camera operation by the application layer APP, and then convert these parameters into corresponding instructions and send them to the client, and the client will actually complete the operation of the camera.
  • Step 503 The client parses the command to control the camera, and sets and starts the corresponding camera device according to the usage parameters carried in the command to control the camera;
  • Step 504 the camera device in the client collects image frames, and stores the collected image frames into a cache queue;
  • the normally started camera device will output the image data of each frame according to the set frame rate, and the client will regularly obtain these image frames and store them in the cache queue.
  • the camera management module When storing the image frame into the frame buffer queue, the camera management module will judge the amount of data accumulated in the current buffer queue, and will clear the expired data (ie, the above-mentioned target image frame) in the case of a large amount of accumulated data.
  • the frame rate of the camera will be reduced to ensure the real-time performance of the camera image after the camera is redirected.
  • Step 505 the H264 encoding module of the client obtains the image frame from the cache queue, and sends it to the mobile phone virtual machine after H264 encoding;
  • Step 506 The mobile phone virtual machine receives the encoded image frame, and then injects it into the camera hardware abstraction layer after H264 decoding;
  • the decoded data is injected into the camera hardware abstraction layer in the mobile phone virtual machine operating system, and the APP running in the mobile phone virtual machine can display the camera image normally.
  • Step 507 The virtual machine of the mobile phone detects the camera shutdown request, sends a stop command to the client, and the client closes the camera after receiving the stop command, and removes the established link at the same time.
  • the camera hardware abstraction layer of the mobile phone virtual machine can also capture the APP's request to turn off the camera, and through the interface machine Server, the command to turn off the camera can also be sent to the client, and the client will turn off the camera after receiving the turn off command. Remove the camera redirection data link with the mobile virtual machine.
  • the camera management module is introduced.
  • the camera management module is responsible for clearing the expired data in the cache queue in time. If it cannot be sent in time, the frame rate of the camera will be adjusted to avoid the accumulation of image frames, resulting in a large delay in the camera screen after the camera is redirected, so as to ensure the real-time performance of the camera screen.
  • the image frame is encoded by H264 encoding, which reduces a large amount of redundant information in the image frame, reduces the amount of data to be sent, and improves the stability of the redirected image.
  • this embodiment is a system embodiment corresponding to the above-mentioned embodiment applied to the client and the embodiment applied to the server, and this embodiment can cooperate with the above-mentioned embodiment applied to the client and the embodiment applied to the server implement.
  • the relevant technical details and technical effects mentioned in the above embodiment applied to the client and the embodiment applied to the server are still valid in this embodiment, and will not be repeated here to reduce repetition.
  • the relevant technical details mentioned in this embodiment can also be applied to the above embodiments applied to the client and the embodiments applied to the server.
  • the embodiment of the present application also provides an electronic device, as shown in FIG. 6 , including: at least one processor 601; and,
  • a memory 602 connected in communication with the at least one processor 601; wherein, the memory 602 stores instructions executable by the at least one processor 601; when the electronic device is a client, the instructions are The at least one processor 601 executes, so that the at least one processor 601 can execute the camera control method applied to the client; when the electronic device is a server, the instructions are executed by the at least one processor 601 Execute, so that the at least one processor 601 can execute the camera control method applied to the server.
  • the memory 602 and the processor 601 are connected by a bus, and the bus may include any number of interconnected buses and bridges, and the bus connects one or more processors 601 and various circuits of the memory 602 together.
  • the bus may also connect together various other circuits such as peripherals, voltage regulators, and power management circuits, all of which are well known in the art and therefore will not be further described herein.
  • the bus interface provides an interface between the bus and the transceivers.
  • a transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing means for communicating with various other devices over a transmission medium.
  • the data processed by the processor 601 is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor 601 .
  • Processor 601 is responsible for managing the bus and general processing, and may also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. And the memory 602 may be used to store data used by the processor 601 when performing operations.
  • the embodiment of the present application also provides a computer-readable storage medium storing a computer program.
  • the above method embodiments are implemented when the computer program is executed by the processor.
  • a storage medium includes several instructions to make a device ( It may be a single-chip microcomputer, a chip, etc.) or a processor (processor) to execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Studio Devices (AREA)

Abstract

本申请实施例涉及互联网技术领域，特别涉及一种摄像头控制方法、电子设备和计算机可读存储介质。摄像头控制方法包括：接收服务器中运行的虚拟终端发送的对摄像头的控制指令；根据所述控制指令，控制所述摄像头采集图像帧，并将所述图像帧存入缓存队列；在所述缓存队列中的图像帧的数量大于预设数量阈值的情况下，清除所述缓存队列中的目标图像帧；其中，所述目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设间隔；将所述缓存队列中剩余的图像帧，发送给所述服务器以供所述服务器将接收到的图像帧发送至所述虚拟终端的操作系统内的相机硬件抽象层。

Description

摄像头控制方法、系统、电子设备和存储介质
相关申请的交叉引用
本申请基于申请号为“2021111316385”、申请日为2021年09月26日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以引入方式并入本申请。
技术领域
本申请实施例涉及互联网技术领域，特别涉及一种摄像头控制方法、电子设备和存储介质。
背景技术
云手机，就是将云计算技术运用于网络终端服务，通过云服务器实现云服务的手机。其实就是深度结合了网络服务的智能手机，这类手机凭借自带的系统以及厂商架设的网络终端可以通过网络实现众多的功能。云手机(也称手机虚机)的操作系统是运行在云端的，没有摄像头可以使用，这就导致手机虚机内的没有相机功能，手机虚机内的应用程序(Application，APP)如果需要使用拍照、录像、扫描二维码等相机相关的功能将无法正常运行。
为解决手机虚机内运行的APP无法操作相机硬件的问题，一般采用相机重定向的方法，即建立手机虚机与客户端之间的连接，将客户端的相机映射给手机虚机使用，但这种方案容易影响摄像头重定向后画面的实时性，从而影响用户体验。
发明内容
本申请实施例提供了一种摄像头控制方法，应用于客户端，包括：接收服务器中运行的虚拟终端发送的对摄像头的控制指令；根据所述控制指令，控制所述摄像头采集图像帧，并将所述图像帧存入缓存队列；在所述缓存队列中的图像帧的数量大于预设数量阈值的情况下，清除所述缓存队列中的目标图像帧；其中，所述目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设间隔；将所述缓存队列中剩余的图像帧，发送给所述服务器以供所述服务器将接收到的图像帧发送至所述虚拟终端的操作系统内的相机硬件抽象层。
本申请实施例还提供了一种摄像头控制方法，应用于服务器，所述服务器中运行有虚拟终端，所述方法包括：向客户端发送对摄像头的控制指令；其中，所述客户端根据所述控制指令，控制所述摄像头采集图像帧，并将所述图像帧存入缓存队列；在所述缓存队列中的图像帧的数量大于预设数量阈值的情况下，清除所述缓存队列中的目标图像帧并将所述缓存队列中剩余的图像帧，发送给所述服务器；其中，所述目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设时间间隔；接收所述客户端发送的图像帧，并将所述图像帧发送至所述虚拟终端的操作系统内的相机硬件抽象层。
本申请实施例还提供了一种摄像头控制系统，包括：服务器和客户端，所述服务器中运行有虚拟终端；所述服务器用于向所述客户端发送对摄像头的控制指令；所述客户端用于根 据所述控制指令，控制所述摄像头采集图像帧，并将所述图像帧存入缓存队列；所述客户端还用于在所述缓存队列中的图像帧的数量大于预设数量阈值的情况下，清除所述缓存队列中的目标图像帧，并将所述缓存队列中剩余的图像帧发送给所述服务器；其中，所述目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设间隔；所述服务器还用于将接收到的图像帧发送至所述虚拟终端的操作系统内的相机硬件抽象层。
本申请实施例还提供了一种电子设备，包括：至少一个处理器；以及，与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述至少一个处理器执行的指令；在所述电子设备为客户端的情况下，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行应用于客户端的摄像头控制方法；在所述电子设备为服务器的情况下，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行应用于服务器的摄像头控制方法。
本申请实施例还提供了一种计算机可读存储介质，存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现应用于客户端的摄像头控制方法，或者实现应用于服务器的摄像头控制方法。
附图说明
图1是本申请实施例中提到的应用场景的示意图；
图2是本申请实施例中提到的一种应用于客户端的摄像头控制方法的流程示意图；
图3是本申请实施例中提到的一种应用于服务器的摄像头控制方法的流程示意图；
图4是本申请实施例中提到的一种摄像头控制系统的示意图；
图5是本申请实施例中提到的另一种摄像头控制方法的流程示意图；
图6是本申请实施例中提到的电子设备的结构示意图。
具体实施方式
本申请实施例的主要目的在于提出一种摄像头控制方法、电子设备和计算机可读存储介质，使得可以提高摄像头重定向后画面的实时性，从而提高用户体验。
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合附图对本申请的各实施例进行详细的阐述。然而，本领域的普通技术人员可以理解，在本申请各实施例中，为了使读者更好地理解本申请而提出了许多技术细节。但是，即使没有这些技术细节和基于以下各实施例的种种变化和修改，也可以实现本申请所要求保护的技术方案。以下各个实施例的划分是为了描述方便，不应对本申请的具体实现方式构成任何限定，各个实施例在不矛盾的前提下可以相互结合相互引用。
为便于对本申请实施例的理解，下面首先对本申请中所涉及的应用场景进行简要说明：
在一个例子中，本申请实施例的应用场景的示意图可以参考图1，云手机系统包括：客户端(比如图1中的客户端1、客户端2、客户端3)、服务器。服务器中可以包括：登陆管理模块、虚机管理模块，服务器中还运行有手机虚机(比如图1中的手机虚机1、手机虚机2、手机虚机3)。
客户端，用于向服务器发起登陆请求，服务器对登陆账号进行验证，验证通过后为相应的登陆用户分配一台手机虚机，并将这台手机虚机的连接参数返回给客户端，客户端可以通过返回的连接参数建立与相应的手机虚机连接的链路，连接成功后通过客户端就可以操作相应的手机虚机。
服务端，运行有登陆管理模块、虚机管理模块等。登陆管理模块用于管理用户信息和登陆验证；虚机管理模块用于对手机虚机进行管理，包括手机虚机的创建、删除、启动、停止等，还可以用于管理手机虚机的运行状态。
手机虚机为运行在服务器中的独立系统，客户端可以与之建立连接，并通过建立的链路对手机虚机进行操作。通常手机虚机上还可以部署一些预设的模块，通过这些模块可以对手机虚机进行管控。
本申请的实施例提供了一种摄像头控制方法，应用于客户端。该客户端可以为图1的云手机系统中的客户端，客户端可以为普通的手机设备，但是使用云手机时所述客户端可以只起到人机交互的功能。客户端上设有摄像头(即相机设备)，客户端的数量可以是多个，如图l所示，服务器可以同时连接客户端1、客户端2和客户端3，服务器上可以同时运行多个手机虚机的操作系统，分别供客户端1、客户端2和客户端3使用。本实施例中的摄像头控制方法的流程示意图可以参考图2，包括：
步骤201：接收服务器中运行的虚拟终端发送的对摄像头的控制指令；
步骤202：根据控制指令，控制摄像头采集图像帧，并将图像帧存入缓存队列；
步骤203：在缓存队列中的图像帧的数量大于预设数量阈值的情况下，清除缓存队列中的目标图像帧；
步骤204：将缓存队列中剩余的图像帧，发送给服务器以供服务器将接收到的图像帧发送至虚拟终端的操作系统内的相机硬件抽象层。
本实施例中，当处于缓存队列中的图像帧的数量大于预设数量阈值的情况下，说明当前网络状态不佳导致缓存队列中当前存在大量图像帧无法及时上报，即缓存队列中堆积了大量数据，此时客户端会清除缓存队列中的目标图像帧，目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设间隔，也就是说目标图像帧是很早之前存入缓存队列中的图像帧，可以视为过期数据，即本申请实施例中可以在缓存队列中堆积大量图像帧的情况下，及时清除缓存队列中的过期数据。然后，客户端将缓存队列中剩余的图像帧，发送给服务器以供服务器将接收到的图像帧发送至虚拟终端的操作系统内的相机硬件抽象层。也就是说，客户端可以及时将较新的图像帧发送给服务器，从而服务器将这些较新的图像帧发送至虚拟终端的操作系统内的相机硬件抽象层，避免缓存队列中的图像帧的大量堆积导致摄像头重定向后图像画面出现较大延时，有利于提高摄像头重定向后画面的实时性，从而提高用户体验。
在步骤201中，客户端可以接收服务器中运行的虚拟终端发送的对摄像头的控制指令。其中，服务器中运行的虚拟终端可以为图1中的手机虚机。手机虚机的操作系统中的相机硬件抽象层可以拦截到手机虚机内运行的APP对相机的使用请求(也可以理解为对摄像头的使用请求)，该使用请求中可以携带使用参数，使用参数可以包括APP对相机的使用时期望的相机参数，比如可以包括：摄像头分辨率、闪光灯是否开启、拍摄模式(比如人像模式、风景模式、夜景模式等)、拍摄图像的存储地址、开启前置摄像头还是后置摄像头、帧率等。相机硬件抽象层可以根据携带使用参数的使用请求生成对摄像头的控制指令，并将该摄像头 的控制指令发送给客户端。
在一个例子中，客户端与服务器之间建立有传输控制协议(Transmission Control Protocol，TCP)链路，客户端与服务器之间可以通过该TCP链路传输数据，比如服务端可以通过该TCP链路将对摄像头的控制指令发送给客户端。
在步骤202中，客户端可以根据控制指令，控制摄像头采集图像帧，并将图像帧存入缓存队列。客户端可以根据控制指令获取手机虚机内运行的APP对相机的使用参数，并根据使用参数对客户端内的相机设备即摄像头进行设置，比如设置摄像头分辨率、拍照时闪光灯是否开启、拍摄模式(比如人像模式、风景模式、夜景模式等)、拍摄图像的存储地址、开启前置摄像头还是后置摄像头等。在对客户端内的相机设备即摄像头完成设置后，可以进行图像捕获即采集图像帧。客户端可以将采集的图像帧均存入预设的缓存队列中，存入缓存队列中的图像帧可以均对应有时间戳，图像帧的时间戳可以理解为采集到该图像帧的时间点，也可以理解为将采集的图像帧存入缓存队列中的时间点。
在步骤203中，目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设间隔。客户端可以在将采集的图像帧存入缓存队列后，判断缓存队列中存入的图像帧的数量是否大于预设数量阈值，并在判定缓存队列中存入的图像帧的数量大于预设数量阈值的情况下，清除缓存队列中的目标图像帧。其中，预设数量阈值可以根据实际需要进行设置，用于表征当前缓存队列中存入的图像帧的数量过多。本申请的发明人发现，当网络状态不佳时，缓存队列中的图像帧可能难以被及时发送至服务器，因此容易在缓存队列中堆积大量图像帧，服务器不能及时接收到最新的图像帧，容易造成画面的延时。因此，本实施例通过清除缓存队列中的目标图像帧，即清除缓存队列中较老的数据，以避免缓存队列中的数据堆积，使得排列在缓存队列的队首的图像帧为相对较新的数据。
在步骤204中，相机硬件抽象层(Hardware Abstraction Layer，HAL)位于操作系统内核与硬件电路之间的接口层。客户端将缓存队列中剩余的图像帧，发送给服务器，从而服务器会将接收到的图像帧发送至虚拟终端的操作系统内的相机硬件抽象层。比如，客户端可以按照各剩余的图像帧进入缓存队列的顺序，依次将各剩余的图像帧发送给服务器。
在具体实现中，客户端可以通过与服务器建立的TCP链路将缓存队列中剩余的图像帧，发送给服务器。服务器可以将接收到的图像帧发送给发起对摄像头的控制指令的虚拟终端的操作系统内的相机硬件抽象层，则运行在该虚拟终端中的请求使用摄像头的APP就可以正常显示相机画面了。
在一个实施例中，在清除缓存队列中的目标图像帧之后，还包括：在缓存队列中存在超过预设时长未被发送给所述服务器的图像帧的情况下，降低摄像头的帧率。也就是说，清除缓存队列中的目标图像帧之后，如果检测到缓存队列中依然存在超过预设时长未被发送给服务器的图像帧的情况，说明网络不佳的状态不是偶尔出现或是短时间出现的，而是当前一段时间内不可避免的客观情况，此时可以降低摄像头的帧率，即降低每秒显示的图像帧的数量，以适应当前网络不佳的客观情况，保证用户有较好的使用体验。
在一个例子中，降低所述摄像头的帧率，包括：确定向所述服务器发送所述缓存队列中的图像帧的发送速度；根据所述发送速度确定帧率调整量，并根据所述帧率调整量降低所述摄像头的帧率。其中，向服务器发送图像帧的速度越快，说明当前的网速相对较快，网络状 态相对较好，向服务器发送图像帧的速度越慢，说明当前的网速相对较慢，网络状态相对较差。向服务器发送图像帧的速度越小，帧率调整量越大，向服务器发送图像帧的速度越大，帧率调整量越小，降低后的摄像头的帧率为摄像头的当前帧率减去帧率调整量。也就是说，向服务器发送图像帧的速度越小，降低后的帧率越小，向服务器发送图像帧的速度越大，降低后的帧率越大。可选的，客户端中可以预存发送速度与帧率调整量的对应关系，从而客户端可以根据预存的该对应关系和向服务器发送图像帧的发送速度，确定帧率调整量。通过这种帧率调整方式可以更好的适应不同的网络状态。
在另一个例子中，降低所述摄像头的帧率也可以通过如下方式实现；以预设的步进值降低摄像头的帧率；其中，步进值可以根据实际需要进行设置，即每次将摄像头的帧率降低步进值的大小。
在一个例子中，在降低所述摄像头的帧率时，所述帧率最低被降低到预设帧率。也就是说，帧率不会无限制的降低，当帧率为预设帧率时，不会再对帧率进行进一步的降低。通过设置预设帧率使得可以在降低帧率以适应当前网络不佳的客观情况的同时控制帧率降低的大小，不至于使降低后的帧率过小，从而可以在一定程度上保证画面显示的连贯性。
其中，预设帧率可以根据实际需要进行设置，比如根据用户对连贯性的要求进行设定，如果对画面连贯性的要求高，则预设帧率可以设置的相对较大，反之，如果对画面连贯性的要求低，则预设帧率可以设置的相对较小。在具体实现中，预设帧率可以设置为20fps，即帧率最低被降低到20fps。
在一个实施例中，将所述缓存队列中剩余的图像帧，发送给所述服务器中运行的所述虚拟终端即上述的步骤204，包括：将所述缓存队列中剩余的图像帧进行H264编码，得到编码后的图像帧；将所述编码后的图像帧，发送给所述服务器中运行的所述虚拟终端，以供所述虚拟终端对所述编码后的图像帧进行H264解码得到解码后的图像帧。通过对图像帧进行H264编码，可以降低图像帧数据中的大量冗余信息，减少需要发送的数据量，从而加快客户端向服务器发送图像帧的速度，在一定程度上有利于提高相机画面的实时性。
其中，编码可以为软编码也可以为硬编码，软编码利用中央处理器(Central Processing Unit，CPU)的运算完成，硬编码利用图形处理器(Graphics Processing Unit，GPU)的运算完成，硬编码的效率远高于软编码。对应的解码可以为软解码也可以为硬解码，软解码利用CPU的运算完成，硬解码利用GPU的运算完成，硬解码的效率远高于软解码。
本申请的实施例还提供了一种摄像头控制方法，应用于服务器，服务器中运行有虚拟终端，该服务器可以为图1中的服务器，应用于服务器的摄像头控制方法的流程示意图可以参考图3，包括：
步骤301：向客户端发送对摄像头的控制指令；
其中，客户端根据控制指令，控制摄像头采集图像帧，并将图像帧存入缓存队列；在缓存队列中的图像帧的数量大于预设数量阈值的情况下，清除缓存队列中的目标图像帧并将缓存队列中剩余的图像帧，发送给服务器；其中，目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设间隔。
在具体实现中，当服务器检测到虚拟终端调用摄像头的请求(即对摄像头的使用请求)， 可以根据调用摄像头的请求，向客户端发送对摄像头的控制指令。虚拟终端调用摄像头的请求可以具体为虚拟终端中运行的某个APP调用摄像头的请求。
步骤302：接收客户端发送的图像帧，并将图像帧发送至虚拟终端的操作系统内的相机硬件抽象层。
在一个例子中，接收所述客户端发送的图像帧，包括：接收所述客户端发送的编码后的图像帧；其中，客户端对缓存队列中剩余的图像帧进行H264编码，得到编码后的图像帧；对所述编码后的图像帧进行H264解码，得到解码后的图像帧；将所述解码后的图像帧发送至所述虚拟终端的操作系统内的相机硬件抽象层。
需要说明的是，本申请实施例中的上述各示例均为为方便理解进行的举例说明，并不对本申请的技术方案构成限定。
不难发现，本实施例为与上述应用于客户端的实施例对应的应用于服务器的实施例，本实施例可与上述应用于客户端的实施例互相配合实施。上述应用于客户端的实施例中提到的相关技术细节和技术效果在本实施例中依然有效，为了减少重复，这里不再赘述。相应地，本实施例中提到的相关技术细节也可应用在上述应用于客户端的实施例中。
上面各种方法的步骤划分，只是为了描述清楚，实现时可以合并为一个步骤或者对某些步骤进行拆分，分解为多个步骤，只要包括相同的逻辑关系，都在本专利的保护范围内；对算法中或者流程中添加无关紧要的修改或者引入无关紧要的设计，但不改变其算法和流程的核心设计都在该专利的保护范围内。
本申请实施例还提供了一种摄像头控制系统，包括：服务器和客户端，所述服务器中运行有虚拟终端；所述服务器用于向所述客户端发送对摄像头的控制指令；所述客户端用于根据所述控制指令，控制所述摄像头采集图像帧，并将所述图像帧存入缓存队列；所述客户端还用于在所述缓存队列中的图像帧的数量大于预设数量阈值的情况下，清除所述缓存队列中的目标图像帧，并将所述缓存队列中剩余的图像帧发送给所述服务器；其中，所述目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设间隔；所述服务器还用于将接收到的图像帧发送至所述虚拟终端的操作系统内的相机硬件抽象层。
在一个例子中，摄像头控制系统的示意图可以参考图4，包括：服务器401和客户端402，服务器401中运行的虚拟终端包括：相机硬件抽象层4010、H264解码模块4011、接口机Server4012；客户端402包括：相机设备4020、帧缓存模块4021、相机控制模块4022、H264编码模块4023、接口机Client4024。
相机硬件抽象层4010，用于拦截获取到虚拟终端(比如手机虚机内)运行的APP对相机的使用请求，该使用请求中携带使用参数，相机硬件抽象层4010将该使用请求转换为控制相机的指令(也可以称为对摄像头的控制指令)，并通过运行在手机虚机内的接口机Server4012发送到客户端402。在具体实现中，服务器401在检测到有APP对相机的使用请求时，可以通过客户端402与手机虚机之间已经存在的链路发送通知到手机虚机，然后由接口机Client4024建立与接口机Server4012之间的相机重定向数据链路。
接口机Server4012，用于下发控制相机的指令，并接收客户端发送的图像帧。
H264解码模块4011，用于对接口机Server4012接收到的编码后的图像帧进行解码，将解码后的数据注入相机硬件抽象层4010。
接口机Client4024，用于建立与接口机Server4012的TCP链路，接收接口机Server4012发送的控制相机的指令，并将H264编码后的图像帧发送到服务器。
相机设备4020，用于根据控制相机的指令完成相机设置和图像捕获，并将捕获到的图像帧存入帧缓存模块4021的缓存队列中。其中，相机设备即可以理解为摄像头。
帧缓存模块4021，用于管理和缓存图像帧，在网络不佳时缓存队列中的图像帧数量较多(即上述的判定缓存队列中存入的图像帧的数量大于预设数量阈值)，则将缓存队列中较老的数据(即上述的目标图像帧)丢弃，以保证相机视频的实时性。
相机控制模块4022，用于根据当前网络状况对相机设备4020和帧缓存模块4021进行实时管理。在网络不佳的情况下，缓存队列中将会堆积大量数据，导致服务器不能及时收到最新的帧数据出现画面有延时，相机控制模块4022对缓存队列中的数据量进行检测，当缓存的图像帧的数量较大时及时清除缓存队列中较老的数据(即过期数据)。当缓存队列中的图像帧依然长时间无法及时发送，将降低相机帧率，避免图像帧的大量堆积，保证摄像头重定向后画面的实时性。
H264编码模块4023，用于从帧缓存模块4021中获取图像帧，并进行H264编码，并将编码后的数据通过接口机Client4024发送到服务器。
在一个例子中，在图4所示的摄像头控制系统下，摄像头的控制方法可以参考图5，包括：
步骤501：手机虚机中的相机硬件抽象层检测到APP对相机的使用请求，通知客户端建立与服务器中运行的手机虚拟之间的相机重定向数据链路；
比如，通过在相机硬件抽象层增加拦截功能，在检测到APP对相机的使用请求时，即通过客户端与手机虚机的连接链路发送通知到客户端，由接口机Client建立与接口机Server之间的相机重定向数据链路。
步骤502：手机虚机中的相机硬件抽象层根据APP对相机的使用请求生成控制相机的指令，并通过所建立的链路下发相机启动命令给客户端；
其中，控制相机的指令中携带有对相机的使用参数。相机硬件抽象层也可以获取应用层APP对相机操作的参数，再将这些参数转化为对应的指令下发到客户端，由客户端实际完成对相机的操作。
步骤503：客户端解析控制相机的指令，并根据控制相机的指令中携带的使用参数设置并启动相应的相机设备；
步骤504：客户端中的相机设备采集图像帧，并将采集的图像帧存入缓存队列；
其中，正常启动的相机设备会根据设置的帧率输出每帧的画面数据，客户端中将会定时获取这些图像帧，并存入缓存队列中。在将图像帧存入帧缓存队列时，相机管理模块会判断当前缓存队列中堆积的数据量大小，在堆积的数据量较多的情况下将清除过期数据(即上述的目标图像帧)。当图像帧长期不能及时发送还将降低相机帧率，保证摄像头重定向后相机画面的实时性。
步骤505：客户端的H264编码模块从缓存队列中获取图像帧，并经过H264编码后发送给手机虚机；
步骤506：手机虚机接收编码后的图像帧，然后经过H264解码后注入相机硬件抽象层；
其中，解码后的数据注入到手机虚机操作系统内的相机硬件抽象层，运行在手机虚机内的APP就可以正常显示相机画面了。
步骤507：手机虚机检测到相机关闭请求，发送停止命令到客户端，客户端收到停止命令后关闭相机，同时拆除所建立的链路。
其中，在手机虚机的相机硬件抽象层也可以捕获到APP关闭相机的请求，通过接口机Server也可以把关闭相机的指令下发给客户端，客户端收到关闭指令后即关闭相机，同时拆除与手机虚机之间的相机重定向数据链路。
本实施例中，第一，引入相机管理模块，在网络不佳，大量图像帧无法及时上报，缓存队列中堆积大量图像帧，相机管理模块负责及时清除缓存队列中过期数据，如果图像帧长时间无法及时发送还将调整相机帧率，避免图像帧的堆积，导致摄像头重定向后相机画面出现较大延时，保证相机画面的实时性。第二，通过H264编码对图像帧进行编码，降低图像帧中的大量冗余信息，减少需要发送的数据量，提高了重定向画面的稳定性。通过以上两项改进，能减少发送所占用的带宽，在网络不佳的情况下仍然可以保证有较好的用户体验
不难发现，本实施例为与上述应用于客户端的实施例和应用于服务器的实施例对应的系统实施例，本实施例可与上述应用于客户端的实施例和应用于服务器的实施例互相配合实施。上述应用于客户端的实施例和应用于服务器的实施例中提到的相关技术细节和技术效果在本实施例中依然有效，为了减少重复，这里不再赘述。相应地，本实施例中提到的相关技术细节也可应用在上述应用于客户端的实施例和应用于服务器的实施例中。
本申请实施例还提供了一种电子设备，如图6所示，包括：至少一个处理器601；以及，
与所述至少一个处理器601通信连接的存储器602；其中，所述存储器602存储有可被所述至少一个处理器601执行的指令；在所述电子设备为客户端的情况下，所述指令被所述至少一个处理器601执行，以使所述至少一个处理器601能够执行应用于客户端的摄像头控制方法；在所述电子设备为服务器的情况下，所述指令被所述至少一个处理器601执行，以使所述至少一个处理器601能够执行应用于服务器的摄像头控制方法。
其中，存储器602和处理器601采用总线方式连接，总线可以包括任意数量的互联的总线和桥，总线将一个或多个处理器601和存储器602的各种电路连接在一起。总线还可以将诸如外围设备、稳压器和功率管理电路等之类的各种其他电路连接在一起，这些都是本领域所公知的，因此，本文不再对其进行进一步描述。总线接口在总线和收发机之间提供接口。收发机可以是一个元件，也可以是多个元件，比如多个接收器和发送器，提供用于在传输介质上与各种其他装置通信的单元。经处理器601处理的数据通过天线在无线介质上进行传输，进一步，天线还接收数据并将数据传送给处理器601。
处理器601负责管理总线和通常的处理，还可以提供各种功能，包括定时，外围接口，电压调节、电源管理以及其他控制功能。而存储器602可以被用于存储处理器601在执行操作时所使用的数据。
本申请实施例还提供了一种计算机可读存储介质，存储有计算机程序。计算机程序被处理器执行时实现上述方法实施例。
即，本领域技术人员可以理解，实现上述实施例方法中的全部或部分步骤是可以通过程 序来指令相关的硬件来完成，该程序存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。
本领域的普通技术人员可以理解，上述各实施方式是实现本申请的具体实施例，而在实际应用中，可以在形式上和细节上对其作各种改变，而不偏离本申请的精神和范围。

Claims (10)

  1. 一种摄像头控制方法，应用于客户端，包括：
    接收服务器中运行的虚拟终端发送的对摄像头的控制指令；
    根据所述控制指令，控制所述摄像头采集图像帧，并将所述图像帧存入缓存队列；
    在所述缓存队列中的图像帧的数量大于预设数量阈值的情况下，清除所述缓存队列中的目标图像帧；其中，所述目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设间隔；
    将所述缓存队列中剩余的图像帧，发送给所述服务器以供所述服务器将接收到的图像帧发送至所述虚拟终端的操作系统内的相机硬件抽象层。
  2. 根据权利要求1所述的摄像头控制方法，其中，在所述清除所述缓存队列中的目标图像帧之后，还包括：
    在所述缓存队列中存在超过预设时长未被发送给所述服务器的图像帧的情况下，降低所述摄像头的帧率。
  3. 根据权利要求2所述的摄像头控制方法，其中，所述降低所述摄像头的帧率，包括：
    确定向所述服务器发送所述缓存队列中的图像帧的发送速度；
    根据所述发送速度确定帧率调整量，并根据所述帧率调整量降低所述摄像头的帧率。
  4. 根据权利要求2或3所述的摄像头控制方法，其中，在降低所述摄像头的帧率时，所述帧率最低被降低到预设帧率。
  5. 根据权利要求1至4任一项所述的摄像头控制方法，其中，所述将所述缓存队列中剩余的图像帧，发送给所述服务器中运行的所述虚拟终端，包括：
    将所述缓存队列中剩余的图像帧进行H264编码，得到编码后的图像帧；
    将所述编码后的图像帧，发送给所述服务器中运行的所述虚拟终端，以供所述虚拟终端对所述编码后的图像帧进行H264解码得到解码后的图像帧。
  6. 一种摄像头控制方法，应用于服务器，所述服务器中运行有虚拟终端，所述方法包括：
    向客户端发送对摄像头的控制指令；其中，所述客户端根据所述控制指令，控制所述摄像头采集图像帧，并将所述图像帧存入缓存队列；在所述缓存队列中的图像帧的数量大于预设数量阈值的情况下，清除所述缓存队列中的目标图像帧并将所述缓存队列中剩余的图像帧，发送给所述服务器；其中，所述目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设间隔；
    接收所述客户端发送的图像帧，并将所述图像帧发送至所述虚拟终端的操作系统内的相机硬件抽象层。
  7. 根据权利要求6所述的摄像头控制方法，所述接收所述客户端发送的图像帧，包括：
    接收所述客户端发送的编码后的图像帧；其中，所述客户端对所述缓存队列中剩余的图像帧进行H264编码，得到编码后的图像帧；
    对所述编码后的图像帧进行H264解码，得到解码后的图像帧；
    将所述解码后的图像帧发送至所述虚拟终端的操作系统内的相机硬件抽象层。
  8. 一种摄像头控制系统，包括：服务器和客户端，所述服务器中运行有虚拟终端；
    所述服务器用于向所述客户端发送对摄像头的控制指令；
    所述客户端用于根据所述控制指令，控制所述摄像头采集图像帧，并将所述图像帧存入缓存队列；
    所述客户端还用于在所述缓存队列中的图像帧的数量大于预设数量阈值的情况下，清除所述缓存队列中的目标图像帧，并将所述缓存队列中剩余的图像帧发送给所述服务器；其中，所述目标图像帧的时间戳与当前时间戳之间的时间间隔大于预设间隔；
    所述服务器还用于将接收到的图像帧发送至所述虚拟终端的操作系统内的相机硬件抽象层。
  9. 一种电子设备，包括：至少一个处理器；以及，
    与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述至少一个处理器执行的指令；
    在所述电子设备为客户端的情况下，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如权利要求1至5中任一所述的摄像头控制方法；
    在所述电子设备为服务器的情况下，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如权利要求6至7中任一所述的摄像头控制方法。
  10. 一种计算机可读存储介质，存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1至5中任一所述的摄像头控制方法，或者实现如权利要求6至7中任一所述的摄像头控制方法。
PCT/CN2022/108302 2021-09-26 2022-07-27 摄像头控制方法、系统、电子设备和存储介质 WO2023045551A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111131638.5A CN115883962A (zh) 2021-09-26 2021-09-26 摄像头控制方法、系统、电子设备和存储介质
CN202111131638.5 2021-09-26

Publications (1)

Publication Number Publication Date
WO2023045551A1 true WO2023045551A1 (zh) 2023-03-30

Family

ID=85719692

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/108302 WO2023045551A1 (zh) 2021-09-26 2022-07-27 摄像头控制方法、系统、电子设备和存储介质

Country Status (2)

Country Link
CN (1) CN115883962A (zh)
WO (1) WO2023045551A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117255245A (zh) * 2023-11-17 2023-12-19 深圳市法本信息技术股份有限公司 摄像头匹配方法、装置、终端设备以及存储介质

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116233372B (zh) * 2023-05-10 2023-07-14 北京城建智控科技股份有限公司 一种移动方舱内部的安全监控方法及系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1787481A (zh) * 2004-12-08 2006-06-14 腾讯科技（深圳）有限公司 一种减小数据包播放延时的方法
CN104410583A (zh) * 2014-11-26 2015-03-11 北京邮电大学 基带资源池中的上行信道数据处理方法
CN108206787A (zh) * 2016-12-17 2018-06-26 北京华为数字技术有限公司 一种拥塞避免方法和装置
CN110493203A (zh) * 2019-07-31 2019-11-22 湖南微算互联信息技术有限公司 一种云手机摄像头控制方法、系统、装置和存储介质
CN111917988A (zh) * 2020-08-28 2020-11-10 长沙摩智云计算机科技有限公司 一种云手机的远程摄像头应用方法、系统及介质
US20200366878A1 (en) * 2019-05-17 2020-11-19 Canon Kabushiki Kaisha Image processing apparatus, control method of image processing apparatus, and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1787481A (zh) * 2004-12-08 2006-06-14 腾讯科技（深圳）有限公司 一种减小数据包播放延时的方法
CN104410583A (zh) * 2014-11-26 2015-03-11 北京邮电大学 基带资源池中的上行信道数据处理方法
CN108206787A (zh) * 2016-12-17 2018-06-26 北京华为数字技术有限公司 一种拥塞避免方法和装置
US20200366878A1 (en) * 2019-05-17 2020-11-19 Canon Kabushiki Kaisha Image processing apparatus, control method of image processing apparatus, and storage medium
CN110493203A (zh) * 2019-07-31 2019-11-22 湖南微算互联信息技术有限公司 一种云手机摄像头控制方法、系统、装置和存储介质
CN111917988A (zh) * 2020-08-28 2020-11-10 长沙摩智云计算机科技有限公司 一种云手机的远程摄像头应用方法、系统及介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117255245A (zh) * 2023-11-17 2023-12-19 深圳市法本信息技术股份有限公司 摄像头匹配方法、装置、终端设备以及存储介质
CN117255245B (zh) * 2023-11-17 2024-03-15 深圳市法本信息技术股份有限公司 摄像头匹配方法、装置、终端设备以及存储介质

Also Published As

Publication number Publication date
CN115883962A (zh) 2023-03-31

Similar Documents

Publication Publication Date Title
WO2023045551A1 (zh) 摄像头控制方法、系统、电子设备和存储介质
CN111135569B (zh) 云游戏处理方法、装置、存储介质与电子设备
CN111882626B (zh) 图像处理方法、装置、服务器及介质
WO2017088484A1 (zh) 基于云计算的实时离屏渲染方法、装置及系统
WO2017219896A1 (zh) 视频流的传输方法及装置
WO2019154221A1 (zh) 发送流数据的方法及数据发送设备
US9213521B2 (en) Control method of information processing apparatus and information processing apparatus
TWI606419B (zh) 經由遠端圖形處理服務串流顯像內容的圖形伺服器以及方法
US20150229960A1 (en) Information processing device, method, and terminal device
WO2021129830A1 (zh) 视频传输方法、装置、设备和系统
WO2021000379A1 (zh) 一种网络数据调度方法及边缘节点
EP3800894A1 (en) Method and apparatus for detecting video playing quality
CN104244037A (zh) 媒体内容缓冲
CN113891175B (zh) 直播推流方法、装置及系统
JPWO2014054325A1 (ja) 符号化制御装置および符号化制御方法
CN115834556B (zh) 数据传输方法、系统、设备、存储介质及程序产品
WO2023040825A1 (zh) 媒体信息的传输方法、计算设备及存储介质
US20240031451A1 (en) Pushing content in time windows based on different positions
CN114363643B (zh) 用于智慧高速的大屏实时视频流播放方法以及装置
WO2019192318A1 (zh) 流量平滑方法、服务器及转发设备
US20140099040A1 (en) Image processing device and image processing method
WO2023051166A1 (zh) 云桌面显示方法、终端、云桌面系统、设备及可读介质
CN115278289B (zh) 一种云应用渲染视频帧处理方法与装置
CN115379235A (zh) 基于缓冲池的图像解码方法、装置、可读介质及电子设备
CN114885209A (zh) 一种直播数据处理方法、计算设备及可读存储介质

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 2022871609

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022871609

Country of ref document: EP

Effective date: 20240326