CN112218003B

CN112218003B - Desktop image acquisition method and device and electronic equipment

Info

Publication number: CN112218003B
Application number: CN201910631280.9A
Authority: CN
Inventors: 曲庆昆
Original assignee: Hangzhou Hikvision System Technology Co Ltd
Current assignee: Hangzhou Hikvision System Technology Co Ltd
Priority date: 2019-07-12
Filing date: 2019-07-12
Publication date: 2023-03-10
Anticipated expiration: 2039-07-12
Also published as: CN112218003A

Abstract

The embodiment of the invention provides a desktop image acquisition method and device and electronic equipment. The method comprises the following steps: acquiring configuration information and image resolution of the electronic equipment, wherein the configuration information is used for representing software and hardware environments of the electronic equipment, and the image resolution is the resolution of an image currently displayed by the electronic equipment; selecting a preset image acquisition scheme corresponding to the configuration information and the image resolution from a plurality of preset image acquisition schemes according to a preset corresponding relation to serve as a target image acquisition scheme; and acquiring the image displayed by the electronic equipment by using the target image acquisition scheme to obtain image data. An appropriate scheme can be decided from multiple image acquisition schemes by acquiring configuration information of the electronic equipment and according to a preset corresponding relation, so that the method and the device are suitable for most application scenes.

Description

Desktop image acquisition method and device and electronic equipment

Technical Field

The invention relates to the technical field of computer remote control, in particular to a desktop image acquisition method and device and electronic equipment.

Background

In some application scenarios, an image displayed on the desktop needs to be acquired, for example, an image displayed on the desktop at the controlled end needs to be acquired in a remote control process. In the related art, an image acquisition interface provided by an operating system or a third party (e.g., openGL) may be used for image acquisition, so that there is a certain requirement on software and hardware environments, and the image acquisition interface provided by the operating system and the third party often shields more details, and when the resolution of an image displayed on a desktop is high (e.g., greater than 4K resolution), the image acquisition effect is poor. Namely, in the related art, an image acquisition method applicable to various application scenarios is lacked.

Disclosure of Invention

The embodiment of the invention aims to provide a desktop image acquisition method, a desktop image acquisition device and electronic equipment so as to provide a desktop image acquisition method which can be suitable for most application scenes. The specific technical scheme is as follows:

in a first aspect of the embodiments of the present invention, a desktop image acquisition method is provided, where the method includes:

acquiring configuration information and image resolution of the electronic equipment, wherein the configuration information is used for representing software and hardware environments of the electronic equipment, and the image resolution is the resolution of an image currently displayed by the electronic equipment;

selecting a preset image acquisition scheme corresponding to the configuration information and the image resolution from a plurality of preset image acquisition schemes according to a preset corresponding relation to serve as a target image acquisition scheme;

and acquiring the image displayed by the electronic equipment by using the target image acquisition scheme to obtain image data.

In a possible embodiment, the selecting, according to a preset correspondence, a preset image acquisition scheme corresponding to the configuration information and the image resolution from a plurality of preset image acquisition schemes as a target image acquisition scheme includes:

judging whether the electronic equipment has an independent display card meeting a preset display card condition according to the configuration information to obtain a first judgment result, wherein the preset display card condition is that a driver of the independent display card has a function of displaying an image frame to be displayed from a cache of the independent display card;

judging whether the image resolution is greater than a preset image resolution threshold value or not to obtain a second judgment result;

and selecting a preset image acquisition scheme corresponding to the first judgment result and the second judgment result from a plurality of preset image acquisition schemes as a target image acquisition scheme according to a preset corresponding relation.

In a possible embodiment, in the preset correspondence, if the first determination result indicates that the electronic device has an independent graphics card that meets the preset graphics card condition, and the second determination result indicates that the image resolution is greater than a preset resolution threshold, the corresponding preset image acquisition scheme is as follows:

reading the image frame to be displayed of the electronic equipment from the cache of the independent display card by using the independent display card as a cache image frame;

and coding the cached image frame by using a CPU of the electronic equipment to obtain coded data serving as the acquired image data.

In a possible embodiment, after the reading, by the independent graphics card, the image frame to be displayed by the electronic device from the buffer of the independent graphics card as a buffer image frame, the corresponding preset image capturing scheme further includes:

displaying the cached image frame;

receiving a region selection instruction input aiming at the cache image frame;

determining a cropping area in the cached image frame according to the area selection instruction;

intercepting an image of a clipping area in the cache image frame to obtain a clipping image of the cache image frame;

the encoding the cached image frame by using the CPU of the electronic device to obtain encoded data as the acquired image data includes:

and encoding the clipped image of the cached image frame by using a CPU of the electronic equipment to obtain encoded data serving as acquired image data.

In a possible embodiment, the corresponding preset image capturing scheme further includes:

determining whether the cropping zone of the cached image frame has changed compared to a last cached image frame;

the encoding, by the CPU of the electronic device, the cropped image of the cached image frame to obtain encoded data as acquired image data includes:

if the clipping area of the cached image frame changes compared with the last cached image frame, the CPU of the electronic equipment is utilized to completely encode the clipping image of the cached image frame to obtain encoded data which is used as the acquired image data;

if the cropping area of the cached image frame is not changed compared with the previous cached image frame, a CPU of the electronic equipment is utilized to encode a difference area in the cropping image of the cached image frame to obtain encoded data as acquired image data, wherein the difference area is an area where the cropping image in the cached image frame is different from the cropping image of the previous cached image frame.

In a possible embodiment, in the preset correspondence, if the first determination result indicates that the electronic device has an independent graphics card that meets the preset graphics card condition, and the second determination result indicates that the image resolution is not greater than a preset resolution threshold, the corresponding preset image acquisition scheme is:

and coding the cached image frame by using the GPU of the independent display card to obtain coded data serving as the acquired image data.

In a possible embodiment, the method further comprises:

judging a condition met by an operating system of the electronic equipment according to the configuration information to obtain a third judgment result;

selecting a preset image acquisition scheme corresponding to the first judgment result and the second judgment result from a plurality of preset image acquisition schemes according to a preset corresponding relation as a target image acquisition scheme, wherein the preset image acquisition scheme comprises:

and selecting a preset image acquisition scheme corresponding to the first judgment result, the second judgment result and the third judgment result from a plurality of preset image acquisition schemes as a target image acquisition scheme according to a preset corresponding relation.

In a possible embodiment, in the preset correspondence, if the first determination result indicates that the electronic device does not have an independent graphics card that satisfies the preset graphics card condition, the second determination result indicates that the image resolution is greater than a preset resolution threshold, and the third determination result indicates that the operating system of the electronic device has a mirror module, the corresponding preset image scheme is as follows:

calling the mirror image module to mirror image equipment interface drawing operation of the operating system on a preset off-screen canvas to obtain a mirror image frame;

and coding the mirror image frame by using a CPU of the electronic equipment to obtain coded data serving as the acquired image data.

In a possible embodiment, in the preset correspondence, if the first determination result indicates that the electronic device does not have an independent graphics card that meets the preset graphics card condition, and the second determination result indicates that the image resolution is greater than a preset resolution threshold, and the third determination result indicates that an operating system of the electronic device has a desktop copy interface, the desktop copy interface is used to obtain a display image frame that is being displayed by the electronic device;

and coding the display image frame by using a CPU of the electronic equipment to obtain coded data serving as the acquired image data.

In a second aspect of the embodiments of the present invention, there is provided a desktop image capturing apparatus, including:

a decision module, configured to select, according to a preset correspondence relationship, a preset image acquisition scheme corresponding to received configuration information and a received image resolution from a plurality of preset image acquisition schemes, as a target image acquisition scheme, where the configuration information represents a hardware and/or software environment of the electronic device, and the image resolution is a resolution of an image displayed by the electronic device;

and the execution module is used for acquiring the image displayed by the electronic equipment by using the target image acquisition scheme to obtain image data.

In a possible embodiment, the decision module is specifically configured to determine, according to the configuration information, whether the electronic device has an independent graphics card that meets a preset graphics card condition, so as to obtain a first determination result, where the preset graphics card condition is that a driver of the independent graphics card has a function of displaying an image frame to be displayed from a cache of the independent graphics card;

reading the image frame to be displayed of the electronic equipment from the cache of the independent display card by using the independent display card to serve as a cache image frame;

displaying the cached image frame;

receiving a region selection instruction input for the cached image frame;

and encoding the cut image of the cache image frame by using a CPU of the electronic equipment to obtain encoded data serving as the acquired image data.

In a possible embodiment, in the preset correspondence, if the first determination result indicates that the electronic device has an independent graphics card that meets the preset graphics card condition, and the second determination result indicates that the image resolution is not greater than a preset resolution threshold, the corresponding preset image capturing scheme is:

In a possible embodiment, the decision module is further configured to determine, according to the configuration information, a condition that is met by an operating system of the electronic device, and obtain a third determination result;

the decision module is specifically configured to select, according to a preset correspondence, a preset image acquisition scheme corresponding to the first determination result, the second determination result, and the third determination result from multiple preset image acquisition schemes, as a target image acquisition scheme.

In a possible embodiment, in the preset correspondence, if the first determination result is used to indicate that the electronic device does not have an independent graphics card that meets the preset graphics card condition, and the second determination result indicates that the image resolution is greater than a preset resolution threshold, the third determination result is used to indicate that an operating system of the electronic device has a desktop copy interface, and a display image frame that is being displayed by the electronic device is obtained through the desktop copy interface;

In a third aspect of embodiments of the present invention, there is provided an electronic device, including:

a memory for storing a computer program;

a processor adapted to perform the method steps of any of the above first aspects when executing a program stored in the memory.

In a fourth aspect of embodiments of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, which, when being executed by a processor, carries out the method steps of any one of the above-mentioned first aspects.

According to the desktop image acquisition method, the desktop image acquisition device and the electronic equipment, a proper scheme can be decided from a plurality of image acquisition schemes by acquiring the configuration information of the electronic equipment and according to the preset corresponding relation, so that the desktop image acquisition method, the desktop image acquisition device and the electronic equipment can be suitable for most application scenes. Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic flow chart of a desktop image acquisition method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of an image acquisition scheme according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of another embodiment of an image acquisition scheme;

FIG. 4 is a schematic diagram of another embodiment of an image acquisition scheme;

FIG. 5 is a schematic diagram of another embodiment of an image acquisition scheme;

FIG. 6 is a schematic diagram of another embodiment of an image acquisition scheme;

FIG. 7 is a schematic diagram of another embodiment of an image acquisition scheme;

fig. 8 is a schematic structural diagram of a desktop image capturing device according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.

Referring to fig. 1, fig. 1 is a schematic flow chart of a desktop image capturing method provided in an embodiment of the present invention, where the method is applied to an electronic device with an image capturing function, and the method may include:

s101, obtaining configuration information and image acquisition resolution of the electronic equipment.

The configuration information is used for representing software and hardware environments of the electronic equipment, and the image resolution is the resolution of an image currently displayed by the electronic equipment. For example, the configuration information may be used to indicate a model number of a graphics card used by the electronic device, a manufacturer, a driver of the graphics card, a version of an operating system used, and the like.

S102, selecting a preset image acquisition scheme corresponding to the configuration information and the image resolution from a plurality of preset image acquisition schemes according to the preset corresponding relation to serve as a target image acquisition scheme.

The image acquisition schemes specifically included in the multiple preset image acquisition schemes may be different according to different application scenes, for example, in one possible embodiment, six preset image acquisition schemes may be included, and in another possible embodiment, five preset image acquisition schemes may be included, which is not limited in this embodiment.

In a possible embodiment, it may be determined whether the electronic device has an independent graphics card that meets a preset condition according to the configuration information, and the first determination result is obtained, where the preset graphics card condition is that a driver of the independent graphics card has a function of extracting an image frame to be displayed from a cache of the independent graphics card. And judging whether the image resolution is greater than the preset image resolution to obtain a second judgment result. And selecting a preset image acquisition scheme corresponding to the first judgment result and the second judgment result from a plurality of preset image acquisition schemes as a target image acquisition scheme according to the preset corresponding relation.

In another possible embodiment, it may also be determined whether the electronic device has an independent graphics card that meets a preset condition according to the configuration information, and the first determination result is obtained, where the preset graphics card condition is that a driver of the independent graphics card has a function of extracting an image frame to be displayed from a buffer of the independent graphics card to be displayed. And judging whether the image resolution is greater than the preset image resolution to obtain a second judgment result. And judging the conditions met by the operating system of the electronic equipment according to the configuration information to obtain a third judgment result. And selecting a preset image acquisition scheme corresponding to the first judgment result, the second judgment result and the third judgment result from the plurality of preset image acquisition schemes as a target image acquisition scheme according to the preset corresponding relation.

And S103, acquiring the image displayed by the electronic equipment by using the target image acquisition scheme to obtain image data.

By selecting the embodiment, a proper scheme can be decided from a plurality of image acquisition schemes by acquiring the configuration information of the electronic equipment and according to the preset corresponding relation, so that the method can be suitable for most application scenes

In the corresponding relationship in the desktop image acquisition scheme provided in the embodiment of the present invention, the image acquisition scheme corresponding to various application scenarios will be described below.

When the first determination result indicates that the electronic device has an independent graphics card that meets the preset condition, and the second determination result indicates that the image resolution is greater than the preset resolution threshold, the corresponding image acquisition scheme may be as shown in fig. 2, and include:

and S201, reading the image frame to be displayed of the electronic equipment from the cache of the independent display card as a cache image frame by calling the cache reading module.

The independent display card is a display card with an independent memory, and the independent display card does not need to share the memory of the CPU in the working process, and often has stronger computing capability compared with an integrated display card with the memory shared by the CPU. And because the independent display card has an independent memory, the running of the independent display card does not cause the reduction of the running efficiency of the CPU, and the running of the CPU may be reduced because the integrated display card occupies part of the memory of the CPU in the running process. The buffered image frame may be an image frame to be output, which has been processed by the independent graphics card.

The buffer reading module is a module having a function of reading a buffer image frame from a memory of a graphics card, and illustratively, for example, NVDIACapture SDK (NVFBC module) issued by english viad (NVIDIA) company for Quadro series independent graphics cards and Tesla series independent graphics cards includes an NVFBC module, and the NVFBC module can read the buffer image frame from the buffers of the two series independent graphics cards.

The cache reading module operates on the independent display card, and the independent display card is provided with an independent memory, so that the cache reading module is used for reading the cache image frames from the independent display card and can be independently completed by the independent display card, and the CPU (central processing unit) cannot be provided with an operation load. The buffered image frame may be an unprocessed original image frame stored in the buffer of the independent graphics card, or may be obtained by performing preset image processing (e.g., compression, rendering) on the original image frame read from the buffer of the independent graphics card.

And S202, coding the cached image frame to obtain image data.

The format of the encoding can be set according to actual requirements, for example, the application scene is a recording desktop, and the encoding of the cut image frame into the h.264 or h.265 format can be performed. Also for example, the application scene is desktop image acquisition, and the cut image frame can be coded into a JPEG (Joint Photographic Experts Group) format. The encoding of the cached image frame may be performed by the CPU, for example, by performing soft encoding on the cached image frame by using an encoder preset on the CPU to obtain the screen image, or by the GPU of the independent graphics card, for example, by performing hard encoding on the cached image frame by using the GPU to obtain the screen image, or for example, by performing hard encoding on the cached image frame by using the GPU and an encoder developed based on a CUDA (Unified Device Architecture) platform to obtain the screen image. In the embodiment of the invention, the CPU or the GPU can be selected to be used for completing the coding of the cached image frame according to actual requirements. For example, if the GPU performance of the independent graphics card is high, the GPU may be selected to complete the encoding of the buffered image frames, and if the GPU performance of the independent graphics card is low, the CPU may be selected to complete the encoding of the buffered image frames. In an optional embodiment, the cached image frame may also be determined to be encoded by using a CPU or a GPU based on the resolution of the cached image frame, and further, if the resolution of the cached image frame is not lower than a preset resolution threshold, the cached image frame may be encoded by using the CPU of the electronic device to obtain a screen image; and if the resolution of the cached image frame is lower than the preset resolution threshold, encoding the cached image frame by using the GPU of the independent display card to obtain image data.

Further, in an optional embodiment, the buffered image frames may be completely encoded, and in another optional embodiment, the buffered image frames may also be completely encoded only for an acquired first buffered image frame, for an acquired non-first buffered image frame, an image of a difference region may be extracted from the cropped image frame by using an extraction module, and the extracted image of the difference region may be encoded to obtain a screen image, where the screen image is only a partial image of a picture displayed on a screen, and the difference region is a region of the buffered image frame that has a difference compared with a previous buffered image frame. For example, the cached image frame may be uniformly divided into a plurality of pixel blocks, and the size of the pixel block may be different according to different actual requirements, for example, the size of the pixel block may be 16 × 16, 32 × 32, 64 × 64, or 128 × 128, and in other embodiments, other sizes than the above four sizes may also be used, which is not limited in this embodiment. For convenience of discussion, it is assumed that the buffered image frame has a size of 1920 × 1080, is divided into 2025 pixel blocks having a size of 32 × 32, and the 2025 pixel blocks are respectively denoted as pixel blocks 1 through 2025, and if the buffered image frame is the second buffered image video frame acquired and there is a difference between only the 100 pixel blocks of pixel blocks 1 through 100 compared to the first buffered image video frame acquired, only the pixel blocks 1 through 100 may be encoded, and the pixel blocks 101 through 2025 are not encoded.

It will be appreciated that the picture displayed by the complete screen can be calculated given the last buffered image frame and the image of the difference region. It can be understood that the independent graphics card may also need to bear rendering tasks of a desktop or pictures of some application programs in the electronic device, and under the condition that the desktop resolution of the electronic device is high (such as 4K and above resolutions), the rendering tasks may occupy more computing resources of the independent graphics card, and at this time, if the cut image frames are encoded by using the independent graphics card, encoding may be slow, or the independent graphics card cannot continue to bear the rendering tasks, which may cause a user to pause using the electronic device, and affect user experience. The cropped image frames can be encoded by the CPU to obtain the screen image.

By adopting the embodiment, the cache reading module in the driving program of the independent display card can be called to realize that the independent display card collects the cache image frame, so that the resource consumption of the CPU in the desktop image collecting process is reduced, and the probability of blockage of the electronic equipment caused by desktop image collecting is further reduced.

According to the embodiment, the cache reading module in the driving program of the independent display card is called, the independent display card is used for acquiring the cache image frame, the CPU is used for coding the cut image frame, the resource consumption of the CPU is reduced, the resource consumption of the independent display card and the CPU in the desktop image acquisition process is balanced, the probability of unsmooth electronic equipment due to the desktop image acquisition is reduced, and the desktop image acquisition can still be normally performed under the condition of a high-resolution desktop.

In the application scenario, the electronic device needs to perform desktop image acquisition to obtain a video stream of a desktop and send the video stream to the client, so as to share the desktop to the client. Referring to fig. 3, a flow of desktop image acquisition of the electronic device may include:

and S301, reading the image frame to be displayed of the electronic equipment from the cache of the independent display card as a cache image frame by calling a cache reading module.

The step is the same as S201, and reference may be made to the foregoing description about S201, which is not described herein again.

S302, capturing the image of the preset area in the cache image frame to obtain the cut cache image frame.

The preset region may be determined by displaying a complete buffered image frame, receiving a region selection instruction input for the displayed buffered image frame, and according to the region selection instruction. Wherein the resolution of the buffered image frame after cropping should be lower than the resolution of the buffered image frame before cropping. Further, the size of the preset area may be set by the user according to actual requirements, or may be determined by the electronic device. For example, taking the example that the electronic device outputs the collected desktop to the client device, the size of the preset region may be set to 800 × 600 by the user through the client in advance, or the electronic device acquires the resolution of the main desktop of the client device, and uses the resolution of the main desktop as the size of the preset region, for example, the electronic device may determine the size of the preset region according to the load degree of the current CPU and the independent graphics card, and when the load degree of the CPU and the independent graphics card is lower, it may be indicated that the calculation capability of the CPU and the independent graphics card that are not occupied is stronger, at this time, the size of the preset region may be determined to be a larger value (1920 × 1080), and when the load degree of the CPU and the independent graphics card is higher, it may be indicated that the calculation capability of the CPU and the independent graphics card that are not occupied is weaker, at this time, the size of the preset region may be determined to be a smaller value (800 × 600).

The position of the preset area may be an area of interest selected by a user, or may be a main area in a desktop determined according to a preset rule, for example, a thumbnail of a desktop of the electronic device may be displayed on the client device, the user determines the position of the preset area on the thumbnail by frame selection or dragging a selection frame, or the position of a window which is currently active on the desktop may be determined as the preset area by the electronic device, and when a plurality of windows which are currently active exist on the desktop, the position of a window with the highest preset priority of a corresponding program may be determined as the position of the preset area according to the preset priority of the program corresponding to the windows.

And S303, coding the clipped cache image frame to obtain image data.

In this embodiment, the GPU may intercept the buffered image frame and the CPU encodes the intercepted buffered image frame, or one of the GPU and the CPU may intercept the buffered image frame and encode the intercepted buffered image frame, or the CPU may intercept the buffered image frame and encode the intercepted buffered image frame.

It will be appreciated that encoding a complete buffered image frame may consume more computational resources when the buffered image frame resolution is too high. By adopting the embodiment, the calculation resources consumed for coding the cached image frame can be reduced by intercepting the image of the preset area in the cached image frame. The possibility of jamming of the electronic equipment caused by desktop image acquisition is further reduced.

Referring to fig. 4, fig. 4 is a schematic diagram illustrating another principle of an image capturing scheme according to an embodiment of the present invention, which may include:

s401, reading the image frame to be displayed of the electronic equipment from the cache of the independent display card as a cache image frame by calling a cache reading module.

S402, determining whether the communication quality between the client and the client is higher than a preset communication quality threshold, if the communication quality between the client and the client is higher than the preset communication quality threshold, executing S404, and if the communication quality between the client and the client is not higher than the preset communication quality threshold, executing S403.

In this embodiment, the communication quality with the client may be determined at a data transmission rate with the client, or may be determined at a packet loss rate with the client. The preset communication quality can be set according to actual requirements, or can be determined by the electronic equipment according to preset rules.

And S403, downsampling the cached image frame.

The resolution of the down-sampled buffered image frame is lower relative to the resolution of the buffered image frame before down-sampling, and therefore the amount of data is also less. The parameters of the down-sampling may be set according to actual requirements, for example, assuming that the resolution of the buffered image frame before the down-sampling is 15360 × 4320, when the communication quality is not higher than the preset communication quality threshold, but the difference between the communication quality and the preset communication quality threshold is small, the resolution of the buffered image frame may be reduced to 1920 × 1080 by the down-sampling, and when the communication quality is not higher than the preset communication quality threshold, and the difference between the communication quality and the preset communication quality threshold is large, the resolution of the buffered image frame may be reduced to 600 × 400 by the down-sampling.

S404, coding the cached image frame to obtain image data.

The step is the same as S202, and reference may be made to the description of S202, which is not described herein again.

By adopting the embodiment, the quality of the buffered image frame can be automatically reduced under the condition of poor communication quality, so that the data volume of the first output data is reduced, and the probability of delay caused by the limitation of the communication quality is reduced. The user can remotely control the electronic equipment more smoothly through the client.

Referring to fig. 5, fig. 5 shows another flow chart of a desktop image acquisition method provided by an embodiment of the present invention, which may include:

s501, reading the image frame to be displayed of the electronic equipment from the cache of the independent display card by calling the cache reading module to serve as the cache image frame.

S502, determining whether the resolution of the buffered image frame is higher than a preset resolution threshold, if the resolution of the buffered image frame is not higher than the preset resolution threshold, performing S503, and if the resolution of the buffered image frame is higher than the preset resolution threshold, performing S504.

The preset resolution threshold can be set according to actual requirements, illustratively, if the performance of an independent display card of the electronic device is good, the preset resolution threshold can be set to be high, and if the performance of the independent display card of the electronic device is poor, the preset resolution threshold can be set to be low. In the present embodiment, the preset resolution threshold may be 4K resolution. It is understood that the higher the resolution of the buffered image frames, the more resources are consumed by the rendering tasks that the individual graphics card may be burdened with, and the lower the resolution of the buffered image frames, the less resources are consumed by the rendering tasks that the individual graphics card may be burdened with.

And S503, encoding the cached image frame by using the GPU of the independent display card to obtain a screen image.

For example, assuming that the independent graphics card of the electronic device is a Quadro series independent graphics card or a Tesla series independent graphics card produced by imperial viation corporation, the NVENC module in the independent graphics card driver may be called to encode the cut image frame, and the NVENC module is a hard coding module provided in NVDIA Capture SDK released by imperial viation corporation, and has the functions of h.264 and h.265 format coding.

When the resolution of the cached image frame is lower than the preset resolution threshold, it can be considered that the rendering task required to be borne by the independent graphics card is relatively light, and enough resources are available for encoding the cached image frame. Therefore, in view of further reducing the CPU load, the encoding task may be borne by a separate graphics card. Under the condition, the desktop image acquisition steps can be completed by independent display cards without consuming the resources of the CPU, so that the desktop image acquisition can not cause any influence on the main service of the running software of the CPU.

S504, capturing the image of the preset area in the cache image frame to obtain the cut cache image frame.

The step is the same as S302, and reference may be made to the foregoing description about S302, which is not described herein again.

And S505, coding the cut image frame by using a CPU of the electronic equipment to obtain image data.

By adopting the embodiment, under the condition that the performance of the independent display card is sufficient, the independent display card can completely bear the consumption of computing resources caused by desktop image acquisition, the influence of the desktop image acquisition on a CPU is reduced to the maximum extent, and the probability of blockage caused by the desktop image acquisition can be further reduced.

If the first determination result is used to indicate that the electronic device does not have a graphics card meeting the preset condition, the second determination result indicates that the image resolution is greater than the preset resolution threshold, and the third determination result indicates that the operating system of the electronic device is provided with a mirror module, the corresponding image acquisition scheme may refer to fig. 6, where fig. 6 is another schematic diagram of the image acquisition scheme provided by the embodiment of the present invention, and may include:

s601, calling a mirror image module to mirror the graphic equipment interface drawing operation of the operating system on a preset off-screen canvas to obtain a mirror image frame.

For example, if the electronic device is installed with a Windows XP or Windows 2000 operating system, the Mirror module may be a Mirror Driver module, and if the electronic device is installed with a Windows7 operating system, the Mirror module may be an RDP Encoder Mirror Driver module, or may be a Driver developed by a developer and having a Mirror image graphics device interface drawing operation function.

The mirror image module can be regarded as a virtual driver, and can intercept drawing operations of a Graphics Device Interface (GDI), and the GDI is responsible for information interaction between an operating system and the GDI, so that by intercepting the drawing operations of the GDI and mirroring the operations on the off-screen canvas, images on the off-screen canvas can be consistent with desktop images. It will be appreciated that if no drawing operation is intercepted, the desktop image may be considered unchanged, and it is not necessary to redraw the mirror image frame.

Further, if the installed operating system is windows7 and the used Mirror module is not the RDP Encoder Mirror Driver module, the AERO (adaptive organic reflective Open) special effect of the desktop may be closed, and the rendering effect of the desktop is reduced to improve the rate of the Mirror module drawing the Mirror image frame.

The off-screen canvas is a drawing surface which is not displayed on the current desktop, and if the operating system installed on the electronic device is a Windows series operating system, the intercepted drawing operation on the off-screen canvas can be realized by mirroring through a BitBlt function.

S602, the CPU of the electronic equipment is utilized to encode the mirror image frame to obtain image data.

Some operating systems do not provide a mirror module, but provide a desktop copy interface through which images displayed on the desktop can be obtained. For example, windows 8 and Windows 10 operating systems, have Desktop replication APIs through which images displayed by the Desktop can be copied. Therefore, when the electronic device does not include an independent graphics card, or a driver of the included independent graphics card does not provide a cache reading module, and an operating system installed in the electronic device provides a desktop copy interface, as shown in fig. 7, fig. 7 is another schematic diagram of an image capture scheme provided in an embodiment of the present invention, and the schematic diagram may include:

and S701, acquiring a display image frame through a desktop copying interface.

Taking windows series operating systems as an example, the Desktop copy interface can be Desktop duplicate API, which is a set of standard API provided by Microsoft corporation, provides a collaboration scheme for operating a group of Desktop screenshots by a remote interface, and supports operating systems of windows 8 and above versions.

The Desktop application may create a corresponding device instance through a display program interface, where the display program interface may be Direct 3D (hereinafter referred to as D3D), and D3D is a display program interface developed by microsoft corporation for improving the display performance of a 3D game in a windows series operating system. According to different application scenarios, the display program interface may specifically be D3D11, or may be another version of D3D (for example, the display program interface may be D3D12 in the windows 10 system), and for convenience of discussion, the following description will use the display program interface as D3D11 as an example. The D3D11 device instance may be created through a D3 dcreatededevice interface, and queried through a QueryInterface interface, and finally a Desktop duplicate instance is obtained, which may obtain a Desktop image as a display image frame.

In a case where the display image frames need to be continuously acquired, the display image frames may be completely acquired through the desktop copy interface each time the display image frames are acquired, and in an alternative embodiment, the display image frames may also be completely acquired through the desktop copy interface only when the display image frames are acquired for the first time. When the display image frame is not acquired for the first time, a difference pixel block can be acquired through the desktop copying interface, the difference pixel block is a pixel block which has difference relative to the last acquired display image frame, and the display image frame is drawn based on the difference pixel block and the last acquired display image frame.

Further, the difference pixel block may be an update pixel block in which an image is changed, or a moving pixel block in which a position is changed. For example, if a pixel block is located at position a in the last acquired display image frame, if the pixel block is still located at position a in the currently acquired display image frame, but the image in the pixel block is changed (for example, all included pixel points are changed from blue to green), the pixel block is an update pixel block. If the image in the pixel block is not changed in the display image frame acquired this time, but the pixel block is located at the position B, the pixel block is a moving pixel block.

Illustratively, it is assumed that the display image frame is pre-divided into 16 pixel blocks, which are respectively denoted as pixel blocks 01-16, and the total number of the pixel blocks is 16. Taking the difference pixel block as an update pixel block as an example, assuming that only the images in the pixel block 13 and the pixel block 14 change relative to one display video frame, that is, the pixel block 13 and the pixel block 14 are update pixel blocks, only the changed pixel block 13 and the changed pixel block 14 can be obtained, and on the basis of the previous display video frame, the changed pixel block 13 and the changed pixel block 14 are used to replace the pixel block 13 before the change and the pixel block 14 before the change, so that the current desktop image, that is, the display video frame which needs to be obtained at this time, can be obtained. Taking the difference pixel block as an example of a moving pixel block, assuming that only the position of the pixel block 13 changes with respect to the previous display video frame, that is, the pixel block 13 is a moving pixel block (if the Desktop replication interface is a Desktop replication API, the moving pixel block may be obtained through a getframemotion vectors interface), only the pixel block 13 may be obtained, on the basis of the previous display video frame, a region corresponding to the pixel block 13 in the previous display video frame is used as a failure region, a region corresponding to the pixel block 13 after the movement is used as a migration region, and the image in the pixel block 13 is entirely migrated to the migration region, so as to obtain the current Desktop image. By adopting the embodiment, when the display image frame is not acquired for the first time, the display image frame does not need to be acquired completely, the data quantity required to be acquired is reduced, the rate of acquiring the display image frame is improved, and the desktop image acquisition efficiency is further improved.

S702, the CPU of the electronic equipment is utilized to encode the display image frame to obtain image data.

Referring to fig. 8, fig. 8 is a schematic structural diagram of a desktop image capturing apparatus according to an embodiment of the present invention, which may include:

a decision module 801, configured to select, according to a preset correspondence relationship, a preset image acquisition scheme corresponding to received configuration information and a received image resolution from multiple preset image acquisition schemes, as a target image acquisition scheme, where the configuration information represents a hardware and/or software environment of an electronic device, and the image resolution is a resolution of an image displayed by the electronic device;

the execution module 802 is configured to acquire an image displayed by the electronic device by using a target image acquisition scheme, so as to obtain image data.

In a possible embodiment, the decision module 801 is specifically configured to determine, according to the configuration information, whether the electronic device has an independent graphics card that meets a preset graphics card condition, so as to obtain a first determination result, where the preset graphics card condition is that a driver of the independent graphics card has a function of an image frame to be displayed from a cache of the independent graphics card;

and selecting a preset image acquisition scheme corresponding to the first judgment result and the second judgment result from a plurality of preset image acquisition schemes as a target image acquisition scheme according to the preset corresponding relation.

In a possible embodiment, in the preset correspondence, if the first determination result indicates that the electronic device has an independent graphics card that meets the preset graphics card condition, and the second determination result indicates that the image resolution is greater than the preset resolution threshold, the corresponding preset image acquisition scheme is:

reading an image frame to be displayed of the electronic equipment from a cache of the independent display card by using the independent display card to serve as a cache image frame;

In a possible embodiment, after reading, by using an independent graphics card, an image frame to be displayed by an electronic device from a buffer of the independent graphics card as a buffer image frame, the corresponding preset image capturing scheme further includes:

displaying the cached image frame;

receiving a region selection instruction input aiming at a cache image frame;

determining a clipping region in the cached image frame according to the region selection instruction;

intercepting an image of a clipping area in the cached image frame to obtain a clipped image of the cached image frame;

the method for coding the cached image frame by utilizing the CPU of the electronic equipment to obtain coded data as the collected image data comprises the following steps:

and coding the cut image of the cached image frame by using a CPU of the electronic equipment to obtain coded data serving as the acquired image data.

determining whether a cropping zone of the cached image frame changes as compared to a previous cached image frame;

the method for encoding the cut image of the cache image frame by using the CPU of the electronic equipment to obtain encoded data as the collected image data comprises the following steps:

if the clipping area of the cached image frame changes compared with the last cached image frame, the CPU of the electronic equipment is used for completely coding the clipping image of the cached image frame to obtain coded data which is used as the collected image data;

if the clipping area of the cache image frame is not changed compared with the last cache image frame, a CPU of the electronic equipment is utilized to encode a difference area in the clipping image of the cache image frame to obtain encoded data which is used as the acquired image data, and the difference area is an area where the clipping image in the cache image frame is different from the clipping image of the last cache image frame.

In a possible embodiment, in the preset correspondence, if the first determination result indicates that the electronic device has an independent graphics card that meets the preset graphics card condition, and the second determination result indicates that the image resolution is not greater than the preset resolution threshold, the corresponding preset image acquisition scheme is:

In a possible embodiment, the decision module 801 is further configured to determine, according to the configuration information, a condition that is met by an operating system of the electronic device, and obtain a third determination result;

and the decision module is specifically used for selecting a preset image acquisition scheme corresponding to the first judgment result, the second judgment result and the third judgment result from the plurality of preset image acquisition schemes according to the preset corresponding relation to serve as a target image acquisition scheme.

In a possible embodiment, in the preset correspondence, if the first determination result indicates that the electronic device does not have an independent graphics card satisfying a preset graphics card condition, the second determination result indicates that the image resolution is greater than a preset resolution threshold, and the third determination result indicates that the operating system of the electronic device has a mirror module, the corresponding preset image scheme is as follows:

calling a mirror image module to perform drawing operation on an image equipment interface of a preset off-screen canvas mirror image operation system to obtain a mirror image frame;

In a possible embodiment, in the preset corresponding relationship, if the first determination result is used to indicate that the electronic device does not have an independent graphics card meeting a preset graphics card condition, and the second determination result indicates that the image resolution is greater than a preset resolution threshold, and the third determination result is used to indicate that an operating system of the electronic device has a desktop copy interface, a display image frame being displayed by the electronic device is obtained through the desktop copy interface;

and encoding the display image frame by using a CPU of the electronic equipment to obtain encoded data serving as the acquired image data.

An embodiment of the present invention further provides an electronic device, as shown in fig. 9, including:

a memory 901 for storing a computer program;

the processor 902, when executing the program stored in the memory 701, implements the following steps:

In a possible embodiment, selecting a preset image acquisition scheme corresponding to the configuration information and the image resolution from a plurality of preset image acquisition schemes according to a preset corresponding relationship as a target image acquisition scheme includes:

judging whether the electronic equipment has an independent display card meeting a preset display card condition according to the configuration information to obtain a first judgment result, wherein the preset display card condition is that a driving program of the independent display card has a function of extracting an image frame to be displayed from a cache of the independent display card;

In a possible embodiment, in the preset correspondence, if the first determination result indicates that the electronic device has an independent graphics card that meets the preset graphics card condition, and the second determination result indicates that the image resolution is greater than the preset resolution threshold, the corresponding preset image acquisition scheme is as follows:

displaying the cached image frame;

receiving a region selection instruction input for a cached image frame;

the method for encoding the cached image frame by utilizing the CPU of the electronic equipment to obtain encoded data as the acquired image data comprises the following steps:

if the clipping area of the cache image frame does not change compared with the last cache image frame, a CPU of the electronic equipment is used for coding a difference area in the clipping image of the cache image frame to obtain coded data serving as the acquired image data, wherein the difference area is an area where the clipping image in the cache image frame has difference compared with the clipping image of the last cache image frame.

In one possible embodiment, the method further comprises:

judging the conditions met by the operating system of the electronic equipment according to the configuration information to obtain a third judgment result;

selecting a preset image acquisition scheme corresponding to the first judgment result and the second judgment result from a plurality of preset image acquisition schemes according to a preset corresponding relation, and taking the preset image acquisition scheme as a target image acquisition scheme, wherein the preset image acquisition scheme comprises the following steps:

and selecting a preset image acquisition scheme corresponding to the first judgment result, the second judgment result and the third judgment result from the plurality of preset image acquisition schemes as a target image acquisition scheme according to the preset corresponding relation.

In a possible embodiment, in the preset correspondence relationship, if the first determination result indicates that the electronic device does not have an independent graphics card satisfying a preset graphics card condition, the second determination result indicates that the image resolution is greater than a preset resolution threshold, and the third determination result indicates that the operating system of the electronic device has a mirror module, the corresponding preset image scheme is as follows:

In a possible embodiment, in the preset corresponding relationship, if the first determination result indicates that the electronic device does not have an independent graphics card meeting a preset graphics card condition, the second determination result indicates that the image resolution is greater than a preset resolution threshold, and the third determination result indicates that an operating system of the electronic device has a desktop copy interface, a display image frame being displayed by the electronic device is acquired through the desktop copy interface;

The Memory mentioned in the electronic device may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), for example, at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

In another embodiment of the present invention, a computer-readable storage medium is further provided, in which instructions are stored, and when the instructions are executed on a computer, the instructions cause the computer to execute the desktop image capturing method in any one of the above embodiments.

In a further embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the desktop image acquisition method of any of the above embodiments.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), among others.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the embodiments of the apparatus, the electronic device, the computer-readable storage medium, and the computer program product, since they are substantially similar to the method embodiments, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiments.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A desktop image acquisition method is applied to electronic equipment and is characterized by comprising the following steps:

if the first judgment result shows that the electronic equipment has the independent display card meeting the preset display card condition, selecting a preset image acquisition scheme corresponding to the first judgment result and the second judgment result from a plurality of preset image acquisition schemes according to a preset corresponding relation to serve as a target image acquisition scheme;

if the first judgment result shows that the electronic equipment does not have the independent display card meeting the preset display card condition, judging the condition met by an operating system of the electronic equipment according to the configuration information to obtain a third judgment result, wherein the electronic equipment obtains the image to be displayed by the electronic equipment by utilizing the condition met by the operating system; selecting a preset image acquisition scheme corresponding to the first judgment result, the second judgment result and the third judgment result from a plurality of preset image acquisition schemes as a target image acquisition scheme according to a preset corresponding relation;

2. The method according to claim 1, wherein in the preset correspondence, if the first determination result indicates that the electronic device has an independent graphics card that meets the preset graphics card condition, and the second determination result indicates that the image resolution is greater than a preset resolution threshold, the corresponding preset image capturing scheme is:

3. The method according to claim 2, wherein after the reading, by the independent graphics card, the image frame to be displayed by the electronic device from the buffer memory of the independent graphics card as a buffered image frame, the corresponding preset image capturing scheme further comprises:

displaying the cached image frame;

receiving a region selection instruction input aiming at the cache image frame;

4. The method of claim 3, wherein the corresponding preset image acquisition scheme further comprises:

the encoding, by the CPU of the electronic device, the cropped image of the cached image frame to obtain encoded data as the acquired image data includes:

5. The method according to claim 1, wherein in the preset correspondence relationship, if the first determination result indicates that the electronic device has an independent graphics card satisfying the preset graphics card condition, and the second determination result indicates that the image resolution is not greater than a preset resolution threshold, the corresponding preset image capturing scheme is:

6. The method according to claim 1, wherein in the preset correspondence relationship, if the first determination result indicates that the electronic device does not have an independent graphics card satisfying the preset graphics card condition, the second determination result indicates that the image resolution is greater than a preset resolution threshold, and the third determination result indicates that an operating system of the electronic device has a mirror module, the corresponding preset image scheme is:

7. The method according to claim 1, wherein in the preset correspondence relationship, if the first determination result indicates that the electronic device does not have an independent graphics card satisfying the preset graphics card condition, and the second determination result indicates that the image resolution is greater than a preset resolution threshold, and the third determination result indicates that an operating system of the electronic device has a desktop copy interface, a display image frame being displayed by the electronic device is obtained through the desktop copy interface;

8. A desktop image acquisition device is applied to electronic equipment, and is characterized in that the device comprises:

a decision module, configured to select, according to a preset correspondence relationship, a preset image acquisition scheme corresponding to received configuration information and a received image resolution from a plurality of preset image acquisition schemes, as a target image acquisition scheme, where the configuration information represents a hardware and/or software environment of the electronic device, and the image resolution is a resolution of an image displayed by the electronic device; judging whether the electronic equipment has an independent display card meeting a preset display card condition according to the configuration information to obtain a first judgment result, wherein the preset display card condition is that a driving program of the independent display card has a function of extracting an image frame to be displayed from a cache of the independent display card; judging whether the image resolution is greater than a preset image resolution threshold value or not to obtain a second judgment result; if the first judgment result shows that the electronic equipment has the independent display card meeting the preset display card condition, selecting a preset image acquisition scheme corresponding to the first judgment result and the second judgment result from a plurality of preset image acquisition schemes according to a preset corresponding relation to serve as a target image acquisition scheme; if the first judgment result shows that the electronic equipment does not have the independent display card meeting the preset display card condition, judging a condition met by an operating system of the electronic equipment according to the configuration information to obtain a third judgment result, wherein the electronic equipment acquires an image to be displayed by the electronic equipment by using the condition met by the operating system; selecting a preset image acquisition scheme corresponding to the first judgment result, the second judgment result and the third judgment result from a plurality of preset image acquisition schemes as a target image acquisition scheme according to a preset corresponding relation;

9. The apparatus according to claim 8, wherein in the preset correspondence relationship, if the first determination result indicates that the electronic device has an independent graphics card satisfying the preset graphics card condition, and the second determination result indicates that the image resolution is greater than a preset resolution threshold, the corresponding preset image capturing scheme is:

10. The apparatus according to claim 9, wherein after the reading, by the independent graphics card, the image frame to be displayed by the electronic device from the buffer memory of the independent graphics card as a buffered image frame, the corresponding preset image capturing scheme further includes:

displaying the cached image frame;

receiving a region selection instruction input aiming at the cache image frame;

11. The apparatus of claim 10, wherein the corresponding preset image acquisition scheme further comprises:

12. The apparatus according to claim 8, wherein in the preset correspondence relationship, if the first determination result indicates that the electronic device has an independent graphics card satisfying the preset graphics card condition, and the second determination result indicates that the image resolution is not greater than a preset resolution threshold, the corresponding preset image capturing scheme is:

13. The apparatus of claim 8, wherein in the preset correspondence relationship, if the first determination result indicates that the electronic device does not have an independent graphics card satisfying the preset graphics card condition, the second determination result indicates that the image resolution is greater than a preset resolution threshold, and the third determination result indicates that an operating system of the electronic device has a mirror module, the corresponding preset image scheme is:

14. The apparatus according to claim 8, wherein in the preset correspondence, if the first determination result is used to indicate that the electronic device does not have an independent graphics card that satisfies the preset graphics card condition, and the second determination result indicates that the image resolution is greater than a preset resolution threshold, the third determination result is used to indicate that an operating system of the electronic device has a desktop copy interface, and the display image frame being displayed by the electronic device is obtained through the desktop copy interface;

15. An electronic device, comprising:

a memory for storing a computer program;

a processor for implementing the method steps of any of claims 1 to 7 when executing a program stored in the memory.

16. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.