CN114170122A - Image synthesis method, image synthesis device, electronic equipment and storage medium - Google Patents
Image synthesis method, image synthesis device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114170122A CN114170122A CN202111551043.5A CN202111551043A CN114170122A CN 114170122 A CN114170122 A CN 114170122A CN 202111551043 A CN202111551043 A CN 202111551043A CN 114170122 A CN114170122 A CN 114170122A
- Authority
- CN
- China
- Prior art keywords
- frame rate
- image
- foreground
- image information
- foreground image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001308 synthesis method Methods 0.000 title claims abstract description 56
- 230000015572 biosynthetic process Effects 0.000 title claims abstract description 43
- 238000003786 synthesis reaction Methods 0.000 title claims abstract description 43
- 239000002131 composite material Substances 0.000 claims abstract description 36
- 230000002194 synthesizing effect Effects 0.000 claims abstract description 28
- 238000000034 method Methods 0.000 claims abstract description 18
- 230000006870 function Effects 0.000 claims description 32
- 230000015654 memory Effects 0.000 claims description 30
- 238000000605 extraction Methods 0.000 claims description 28
- 238000012545 processing Methods 0.000 claims description 12
- 239000000203 mixture Substances 0.000 claims description 3
- 230000006854 communication Effects 0.000 abstract description 11
- 238000004891 communication Methods 0.000 abstract description 10
- 230000008569 process Effects 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000006748 scratching Methods 0.000 description 3
- 230000002393 scratching effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention relates to the technical field of audio and video communication, in particular to an image synthesis method, an image synthesis device, electronic equipment and a storage medium, wherein the method comprises the steps of acquiring background images acquired at a first frame rate and foreground image information acquired at a second frame rate, wherein the first frame rate is less than the second frame rate; expanding the background image acquired at the first frame rate based on the second frame rate to obtain a background image corresponding to the second frame rate; synthesizing the background image corresponding to the second frame rate and the foreground image information acquired at the second frame rate to determine a first synthesized image; the first composite image is transmitted. For background images and foreground image information acquired at different frame rates, the background images acquired at a lower acquisition frame rate are expanded to enable the expanded background images to correspond to the second frame rate, so that the background images and the foreground image information with the same acquisition frame rate can be obtained, the background images and the foreground image information are synthesized and then transmitted, and the display synchronization of the background images and the foreground image information at a receiving end can be ensured.
Description
Technical Field
The invention relates to the technical field of audio and video communication, in particular to an image synthesis method and device, electronic equipment and a storage medium.
Background
In real-time audio/video communication processes such as video conferences and remote education, the whole screen (or a certain application program window) of a computer is often required to be shared to participants, such as playing documents or lectures. In the screen sharing process, in addition to sharing a background image such as a played document or a lecture, a changed foreground image, such as a mouse, handwriting, etc., needs to be transmitted to the participants in real time. At present, the conventional method is to collect the mouse information while collecting the screen image, and to superimpose the two together to send out as a frame of image. When the images are continuously collected and transmitted, dynamic video is formed.
If the acquisition frequency is high (for example, 30 frames/second), high requirements are placed on the performance of computer screen acquisition and video coding and the bandwidth of network transmission. For example, a desktop resolution of 1080p requires at least a network transmission bandwidth of 4-8 Mbps for ensuring clear text at 30 frames/second. Therefore, many practical applications reduce the frame rate of screen acquisition to 5 frames/second. Since screen sharing is mainly used for viewing background images such as documents and lectures, the frame rate is not so low because the screen change is not so large. However, if the foreground image on the screen is also transmitted at 5 frames/second, the position of the foreground image is obviously sensed to jump and shake, which brings very serious discomfort to the vision.
Disclosure of Invention
In view of this, embodiments of the present invention provide an image synthesis method, an image synthesis apparatus, an electronic device, and a storage medium, so as to solve the problem of displaying a background image and a foreground image due to different acquisition frame rates.
According to a first aspect, an embodiment of the present invention provides an image synthesis method, including:
acquiring a background image acquired at a first frame rate and foreground image information acquired at a second frame rate, wherein the first frame rate is less than the second frame rate;
expanding the background image acquired at the first frame rate based on the second frame rate to obtain a background image corresponding to the second frame rate;
synthesizing the background image corresponding to the second frame rate and the foreground image information acquired at the second frame rate to determine a first synthesized image;
and sending the first composite image.
According to the image synthesis method provided by the embodiment of the invention, for the background image and the foreground image information acquired at different frame rates, the background image acquired at a smaller acquisition frame rate is expanded, so that the expanded background image corresponds to the second frame rate, the background image and the foreground image information with the same acquisition frame rate can be obtained, and the background image and the foreground image information are synthesized and then transmitted, so that the display synchronization of the background image and the foreground image information at a receiving end can be ensured, and the data transmission quantity is reduced.
With reference to the first aspect, in a first implementation manner of the first aspect, the expanding the background image acquired at the first frame rate based on the second frame rate to obtain a background image corresponding to the second frame rate, where the second frame rate is an integer multiple of the first frame rate includes:
determining an expansion coefficient by using the size relation between the second frame rate and the first frame rate;
and expanding the background image acquired at the first frame rate based on the expansion coefficient, and determining the background image corresponding to the second frame rate.
According to the image synthesis method provided by the embodiment of the invention, the expansion coefficient is determined based on the size relationship between the second frame rate and the first frame rate, so that the accuracy of the expansion coefficient is ensured, and the synchronization between the background image corresponding to the second frame rate and the foreground image information acquired by the second frame rate can be effectively ensured.
According to a second aspect, an embodiment of the present invention further provides an image synthesis method, including:
acquiring a background image acquired at a first frame rate and foreground image information acquired at a second frame rate, wherein the first frame rate is less than the second frame rate;
performing image extraction on the foreground image information acquired at the second frame rate based on the first frame rate to obtain foreground image information corresponding to the first frame rate;
synthesizing the background image acquired at the first frame rate and foreground image information corresponding to the first frame rate to determine a second synthesized image;
and sending the second synthetic image and the foreground image information acquired at the second frame rate.
According to the image synthesis method provided by the embodiment of the invention, for background images and foreground image information acquired at different frame rates, image extraction is carried out on the foreground image information acquired at a larger acquisition frame rate, so that the extracted background images correspond to the first frame rate, background images and foreground image information with the same acquisition frame rate can be obtained, and the foreground image information corresponding to the first frame rate and the background image acquired at the first frame rate are synthesized and then sent, so that the display synchronization of the background images and the foreground image information at a receiving end can be ensured.
With reference to the second aspect, in a first implementation manner of the second aspect, the performing image extraction on foreground image information acquired at a second frame rate based on the first frame rate to obtain foreground image information corresponding to the first frame rate, where the second frame rate is an integer multiple of the first frame rate includes:
determining an extraction coefficient by using the size relation between the second frame rate and the first frame rate;
and extracting the foreground image information acquired at the second frame rate based on the extraction coefficient, and determining the foreground image information at the first frame rate.
According to the image synthesis method provided by the embodiment of the invention, the extraction coefficient is determined based on the size relationship between the second frame rate and the first frame rate, so that the accuracy of the extraction coefficient is ensured, and the synchronization between the foreground image information corresponding to the first frame rate and the background image acquired by the first frame rate can be effectively ensured.
According to a third aspect, an embodiment of the present invention further provides an image synthesis method, including:
receiving a second synthetic image determined by the image synthesis method according to the second aspect of the present invention or the first embodiment of the second aspect and foreground image information acquired at the second frame rate;
acquiring a local synthesis function;
and processing the second synthetic image and the foreground image information acquired at the second frame rate based on the synthetic function to determine a display image.
According to the image synthesis method provided by the embodiment of the invention, the received image not only comprises the second synthesis image but also comprises the foreground image information acquired at the second frame rate, and the display image is determined through the local synthesis function, so that the current display image can be determined according to the local condition, the second synthesis image can be directly displayed, and the current display image can be combined with the foreground image information acquired at the second frame rate to be displayed on the basis of the second synthesis image, so that the image synthesis method can be suitable for different display terminals.
With reference to the third aspect, in a first implementation manner of the third aspect, when a synthesis function is locally provided, the processing the second synthesized image and the foreground image information acquired at the second frame rate based on the synthesis function to determine a display image includes:
matting out a foreground image in the second synthetic image based on the foreground image information to obtain a target background image corresponding to the first frame rate;
expanding the target background image corresponding to the first frame rate based on the second frame rate to obtain a target background image corresponding to the second frame rate;
and synthesizing the target background image corresponding to the second frame rate and the foreground image information acquired at the second frame rate to determine the display image.
According to the image synthesis method provided by the embodiment of the invention, when the synthesis function is locally provided, the second synthesis image is subjected to the scratching and extension of the foreground image information and the synthesis of the foreground image information acquired at the second frame rate, so that the display image is based on the second frame rate, and the fluency of the display image is improved.
With reference to the first embodiment of the third aspect, in a second embodiment of the third aspect, the matting out a foreground image in the second composite image based on the foreground image information to obtain a target background image corresponding to the first frame rate includes:
based on the position in the foreground image, matting and removing the foreground image from the corresponding position of the second synthetic image to obtain a background image to be processed;
acquiring an original background image covered by the foreground image in the background image acquired at the first frame rate;
and synthesizing the original background image and the corresponding background image to be processed, and determining a target background image corresponding to the first frame rate.
According to the image synthesis method provided by the embodiment of the invention, the original background image covered by the original image in the background image acquired at the first frame rate is synthesized with the background image to be processed obtained after the foreground image is removed, so that the background image acquired at the first frame rate can be formed, and the definition of the target background image corresponding to the first frame rate is improved.
With reference to the third aspect, in a third implementation manner of the third aspect, when a synthesis function is not locally provided, the processing the second synthesized image and the foreground image information acquired at the second frame rate based on the synthesis function to determine a display image includes:
and discarding the foreground image information acquired at the second frame rate, and determining the second composite image as the display image.
The image synthesis method provided by the embodiment of the invention can directly display the second synthesized image for the local area without synthesis function.
According to a fourth aspect, an embodiment of the present invention provides an electronic device, including: the image synthesis apparatus includes a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the image synthesis method described in the first aspect or any one of the embodiments of the first aspect, the second aspect or any one of the embodiments of the second aspect, or any one of the embodiments of the third aspect or the third aspect.
According to a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the first aspect or any one of the implementation manners of the first aspect, or any one of the implementation manners of the second aspect or the second aspect, or any one of the implementation manners of the third aspect or the third aspect.
For corresponding beneficial effects of the electronic device and the computer-readable storage medium, reference may be made to the description of the beneficial effects of the corresponding image synthesis method, which is not repeated herein.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow chart of an image synthesis method according to an embodiment of the present invention;
FIG. 2 is a flow chart of an image synthesis method according to an embodiment of the present invention;
FIG. 3 is a flow chart of an image synthesis method according to an embodiment of the present invention;
FIG. 4 is a flow chart of an image synthesis method according to an embodiment of the present invention;
FIG. 5 is a flow chart of an image synthesis method according to an embodiment of the present invention;
FIG. 6 is a flow chart of an image synthesis method according to an embodiment of the present invention;
FIG. 7 is a flow chart of an image synthesis method according to an embodiment of the present invention;
fig. 8 is a block diagram of the structure of an image synthesizing apparatus according to an embodiment of the present invention;
fig. 9 is a block diagram of the structure of an image synthesizing apparatus according to an embodiment of the present invention;
fig. 10 is a block diagram of the structure of an image synthesizing apparatus according to an embodiment of the present invention;
fig. 11 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
According to the image synthesis method provided by the embodiment of the invention, the background image information and the foreground image information are respectively acquired through different frame rates. The foreground image information includes position, size, shape information and the like of the foreground image, the foreground image can be directly collected, the foreground image can be restored by utilizing the size and the shape information of the foreground image, and the specific forming mode of the foreground image is not limited at all. And the transmitting end carries out expansion or extraction processing on the background image or the foreground image according to different requirements, so that the frame rates of the background image and the foreground image in the image transmitted to the receiving end are consistent. The specific definition of the background image and the foreground image can be performed according to actual requirements. For example, in a video conference, a background image is a shared text or PPT, and a foreground image is a mouse or handwriting, and the like; in live broadcasting, a background image is a main broadcast and a background thereof, and a foreground image is a barrage and the like.
In accordance with an embodiment of the present invention, there is provided an image synthesis method embodiment, it is noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer-executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.
In this embodiment, an image synthesis method is provided, which may be used in an electronic device at a sending end, such as a computer, a mobile phone, a tablet computer, and the like, fig. 1 is a flowchart of the image synthesis method according to an embodiment of the present invention, and as shown in fig. 1, the flowchart includes the following steps:
and S11, acquiring background images acquired at a first frame rate and foreground image information acquired at a second frame rate.
Wherein the first frame rate is less than the second frame rate.
The specific values of the first frame rate and the second frame rate are determined according to actual requirements, and are not limited herein. The electronic device acquires the background image at the first frame rate and acquires the foreground image information at the second frame rate, and as described above, in the display mode, the foreground image is displayed by being superimposed on the background image and does not belong to the background image. Therefore, the electronic device can respectively collect the background image and the foreground image information.
For example, the background image is text displayed on a screen of the electronic device, and the foreground image information is mouse information on the text.
And S12, expanding the background image acquired at the first frame rate based on the second frame rate to obtain a background image corresponding to the second frame rate.
Because the first frame rate and the second frame rate are different, in order to ensure that the foreground image information with the lower frame rate causes the problem that the foreground image information with the higher frame rate is displayed suddenly, the frame rates of the background image and the foreground image information need to be adjusted to be consistent. Based on the above, the electronic device expands the background image acquired at the first frame rate based on the second frame rate, so that the background image corresponding to the second frame rate is obtained after the expansion.
Specifically, the expansion may be to copy the background images, for example, to copy each background image acquired at the first frame rate in sequence, and copy the background images the same number of times; or may be copied for different background images for different times, and the specific copying manner is not limited in any way.
Details about this step will be described later.
And S13, synthesizing the background image corresponding to the second frame rate and the foreground image information acquired at the second frame rate, and determining a first synthesized image.
And after the background image at the first frame rate is expanded, obtaining a background image corresponding to a second frame rate, wherein the frame rate corresponding to the background image is the same as the frame rate corresponding to the foreground image information. And the foreground image information and the background image are in one-to-one correspondence, and the foreground image information is synthesized into the background image by utilizing the position information of the foreground image information, so that the obtained first synthesized image has both the background image and the corresponding foreground image information.
S14, the first composite image is transmitted.
The electronic device sends the first composite image to the receiving end, and accordingly the receiving end can display the first composite image after receiving the first composite image, and the problems of display blockage and the like caused by frame rate inconsistency can be avoided.
In the image synthesis method provided by this embodiment, for the background image and the foreground image information acquired at different frame rates, the background image acquired at a relatively low acquisition frame rate is expanded, so that the expanded background image corresponds to the second frame rate, and thus the background image and the foreground image information at the same acquisition frame rate can be obtained and synthesized and then transmitted.
In this embodiment, an image synthesis method is provided, which can be used in electronic devices, such as computers, mobile phones, tablet computers, and the like, and fig. 2 is a flowchart of an image synthesis method according to an embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
and S21, acquiring background images acquired at a first frame rate and foreground image information acquired at a second frame rate.
Wherein the first frame rate is less than the second frame rate.
Please refer to S11 in fig. 1, which is not described herein again.
And S22, expanding the background image acquired at the first frame rate based on the second frame rate to obtain a background image corresponding to the second frame rate.
Specifically, the second frame rate is an integer multiple of the first frame rate, and S22 includes:
s221, determining an expansion coefficient by using the magnitude relation between the second frame rate and the first frame rate.
The electronic device calculates a ratio of the second frame rate to the first frame rate, and determines the ratio as an expansion coefficient. For example, the first frame rate is 5 frames/sec, the second frame rate is 30 frames/sec, and accordingly, the expansion coefficient is 6.
S222, expanding the background image acquired at the first frame rate based on the expansion coefficient, and determining the background image corresponding to the second frame rate.
After the electronic device determines the expansion coefficient, it considers that the background images corresponding to the expansion coefficient in the background images corresponding to the second frame rate obtained after expansion are the same, that is, each frame of background image needs to be copied (expansion coefficient-1) times to obtain the background images corresponding to the expansion coefficient, and thus the background images corresponding to the second frame rate can be obtained. Taking the first frame rate of 5 frames/second and the second frame rate of 30 frames/s as an example, the expansion coefficient is 6, and the background image of each frame needs to be copied 5 times to obtain 6 frames of the same background image.
And S23, synthesizing the background image corresponding to the second frame rate and the foreground image information acquired at the second frame rate, and determining a first synthesized image.
Please refer to S13 in fig. 1, which is not described herein again.
S24, the first composite image is transmitted.
Please refer to S14 in fig. 1, which is not described herein again.
The image synthesis method provided in this embodiment determines the expansion coefficient based on the size relationship between the second frame rate and the first frame rate, so as to ensure the accuracy of the expansion coefficient, and effectively ensure the synchronization between the background image corresponding to the second frame rate and the foreground image information acquired at the second frame rate.
As a specific application example of the embodiment, the first frame rate is 5 frames/second, the second frame rate is 30 frames/s, the background image is a screen image, and the foreground image information is mouse information. As shown in fig. 3, the image synthesis method applied to the information sending end includes:
(1) acquiring a screen image, and 5 frames/second; collecting mouse information for 30 frames/second;
(2) copying the extended screen image to 30 frames/second;
(3) synthesizing the expanded screen image with the mouse information to obtain a synthesized image of 30 frames/second;
(4) sending the composite image of 30 frames/second to a receiving end;
(5) the receiving side displays the 30 frames/sec synthesized image.
The image synthesis method provided by this embodiment can expand the 5 frame/second screen image into 30 frame/second by copying the expanded screen image, and at the same time, the transmission code rate is not increased or decreased. This is because the increased transmission rate is very low because every 6 frames of pictures are identical, the difference between pictures is 0, while video frames are typically encoded as P-frames (encoded according to the difference between pictures). Meanwhile, the compatibility is good, and because the synthesis is completed at the sending end, the problem of mouse jumping can be solved for the receiving end without the synthesis function, and the shared screen image containing the mouse can be normally displayed.
An embodiment of the present invention further provides an image synthesis method, which is applied to an electronic device at a sending end, such as a computer, a mobile phone, a tablet computer, and the like, fig. 4 is a flowchart of the image synthesis method according to the embodiment of the present invention, and as shown in fig. 4, the flowchart includes the following steps:
and S41, acquiring background images acquired at a first frame rate and foreground image information acquired at a second frame rate.
Wherein the first frame rate is less than the second frame rate.
Please refer to S11 in fig. 1, which is not described herein again.
And S42, performing image extraction on the foreground image information acquired at the second frame rate based on the first frame rate to obtain foreground image information corresponding to the first frame rate.
Because the first frame rate and the second frame rate are different, in order to ensure that the foreground image information with the lower frame rate causes the problem that the foreground image information with the higher frame rate is displayed suddenly, the frame rates of the background image and the foreground image information need to be adjusted to be consistent. Based on the above, the electronic device performs image extraction on the foreground image information acquired at the second frame rate based on the first frame rate, so that the foreground image information corresponding to the first frame rate is obtained after extraction.
Specifically, the extraction may be sequentially extracting each foreground image information acquired at the second frame rate, and extracting once every same frame number; the extraction may also be performed based on different frame numbers at different intervals for different foreground image information, and the specific copying manner is not limited at all here.
Details about this step will be described later.
In some optional implementations of this embodiment, the second frame rate is an integer multiple of the first frame rate, and the step S42 may include:
(1) and determining the extraction coefficient by using the size relation between the second frame rate and the first frame rate.
(2) And extracting the foreground image information acquired at the second frame rate based on the extraction coefficient, and determining the foreground image information at the first frame rate.
The electronic device calculates a ratio of the second frame rate to the first frame rate, and determines the ratio as an extraction coefficient. For example, the first frame rate is 5 frames/second, the second frame rate is 30 frames/s, and accordingly, the decimation factor is 6.
After the electronic device determines the extraction coefficient, it needs to extract foreground image information every (expansion coefficient-1) frame to obtain foreground image information corresponding to the first frame rate. Taking the first frame rate as 5 frames/second and the second frame rate as 30 frames/s as an example, the extraction coefficient is 6, and foreground image information corresponding to the first frame rate needs to be extracted once every 5 frames to obtain foreground image information, and accordingly, a foreground image at the first frame rate can be obtained.
The extraction coefficient is determined based on the size relationship between the second frame rate and the first frame rate, so that the accuracy of the extraction coefficient is ensured, and the synchronization between the foreground image corresponding to the first frame rate and the background image acquired by the first frame rate can be effectively ensured.
And S43, synthesizing the background image acquired at the first frame rate and the foreground image information corresponding to the first frame rate, and determining a second synthesized image.
The second composite image is synthesized in a manner similar to that of the first composite image, which can be referred to above specifically, and is not described herein again.
And S44, sending the second composite image and the foreground image information acquired at the second frame rate.
And the electronic equipment sends the second synthetic image and the foreground image information acquired at the second frame rate to the receiving end together, so that the receiving end determines the frame rate of the display image according to the self requirement.
The specific operation on the receiving end will be described in detail below.
In the image synthesis method provided by this embodiment, for background images and foreground image information acquired at different frame rates, image extraction is performed on the foreground image information acquired at a larger acquisition frame rate, so that the extracted background image corresponds to the first frame rate, and thus background images and foreground image information at the same acquisition frame rate can be obtained.
An embodiment of the present invention further provides an image synthesis method, which is applied to an electronic device at a receiving end, such as a computer, a mobile phone, a tablet computer, and the like, fig. 5 is a flowchart of the image synthesis method according to the embodiment of the present invention, and as shown in fig. 5, the flowchart includes the following steps:
and S51, receiving the second synthesized image determined by the image synthesis method and the foreground image information acquired at the second frame rate.
For the second composite image and the foreground image information acquired at the second frame rate, please refer to the related description of the embodiment shown in fig. 4, which is not repeated herein.
And S52, acquiring a local synthesis function.
The local composition function is used to indicate whether the composition function is locally available, which can be obtained from the local attribute information, or determined from the local product version number, etc.
And S53, processing the second composite image and the foreground image information acquired at the second frame rate based on the composite function, and determining a display image.
When the local has a synthesis function, the display image corresponding to the second frame rate is obtained by performing the way of matting, expanding and resynthesis based on the second synthesis image and the forward image acquired at the second frame rate. Details about this process will be described in detail below.
When the local device does not have the synthesis function, the electronic device directly displays the second synthesis image with the first frame rate, and ignores the foreground image information with the second frame rate. Namely, discarding the foreground image information acquired at the second frame rate, and determining the second composite image as the display image.
In the image synthesis method provided by this embodiment, since the received image includes both the second synthesized image and the foreground image information acquired at the second frame rate, and the displayed image is determined by the local synthesis function, the current displayed image can be determined according to the local situation, and the second synthesized image can be directly displayed, and can also be displayed on the basis of the second synthesized image in combination with the foreground image information acquired at the second frame rate, which can be adapted to different display terminals.
An embodiment of the present invention further provides an image synthesis method, which is applied to an electronic device at a receiving end, such as a computer, a mobile phone, a tablet computer, and the like, fig. 6 is a flowchart of the image synthesis method according to the embodiment of the present invention, and as shown in fig. 6, the flowchart includes the following steps:
and S61, receiving the second composite image determined by the image composite method and the foreground image information acquired at the second frame rate.
Please refer to S51 in fig. 5, which is not repeated herein.
And S62, acquiring a local synthesis function.
Please refer to S52 in fig. 5, which is not repeated herein.
And S63, processing the second composite image and the foreground image information acquired at the second frame rate based on the composite function, and determining a display image.
Specifically, S63 includes:
and S631, matting and removing the foreground image in the second composite image based on the foreground image information to obtain a target background image corresponding to the first frame rate.
The electronic equipment determines the position of the foreground image in the second synthetic image by using the position information in the foreground image, determines a minimum bounding box capable of comprising the foreground image based on the determined position, and removes the minimum bounding box to realize the removal of the foreground image; or, based on the shape in the foreground image information, determining the shape outline of the foreground image, and scratching out the foreground image in the second composite image. Of course, other ways to scratch out the foreground image in the second composite image may also be used, and no limitation is made herein. After the foreground image is scratched out, the electronic equipment can obtain a target background image corresponding to the first frame rate. And performing pixel interpolation on the second synthetic image after the foreground image is scratched out to ensure the continuity of the obtained pixels of the target background image corresponding to the second frame rate.
In some optional implementations of this embodiment, the S631 may include:
(1) and based on the position information in the foreground image, matting and removing the foreground image from the corresponding position of the second synthetic image to obtain a background image to be processed.
(2) And acquiring an original background image covered by the foreground image in the background image acquired at the first frame rate.
(3) And synthesizing the original background image and the corresponding background image to be processed, and determining a target background image corresponding to the first frame rate.
The electronic device obtains a background image to be processed after the foreground image is scratched, and for a sending end, when the sending end processes the background image with the first frame rate into a first composite image, the sending end needs to determine the position of the scratched image in the background image with the first frame rate by using foreground image information corresponding to the first frame rate after extraction, and scratches a corresponding background image, namely, an original background image. It should be noted that the original background image is not the entire background image, but a part of the image that is scratched out from the background image. And then synthesizing the corresponding foreground image at the position of the original background image to obtain a first synthesized image. Accordingly, the transmitting end can obtain the original background image with the first frame rate, and transmit the original background image with the first frame rate to the receiving end. Therefore, the receiving end can receive the first composite image at the first frame rate, the original background image at the first frame rate and the foreground image information at the second frame rate, the receiving end removes the foreground image in the first composite image at the first frame rate to obtain the background image to be processed at the first frame rate, and then the original background image at the first frame rate and the background image to be processed at the first frame rate are synthesized to obtain the target background image at the first frame rate.
S632, expanding the target background image corresponding to the first frame rate based on the second frame rate to obtain the target background image corresponding to the second frame rate.
And at the moment, expanding the target background image according to the size relation between the second frame rate and the first frame rate to obtain the target background image corresponding to the second frame rate. For a specific expansion manner, please refer to the related description of S12 or S22 in the embodiment shown in fig. 1 or fig. 2, which is not repeated herein.
And S633, synthesizing the target background image corresponding to the second frame rate and the foreground image information acquired at the second frame rate, and determining a display image.
The specific synthesis process is similar to the synthesis process described above, please refer to the detailed description of S13 in the embodiment shown in fig. 1, and is not repeated herein.
When the image synthesis method provided by this embodiment has a synthesis function locally, the second synthesized image is subjected to the matting and expansion of the foreground image and the synthesis of the foreground image information acquired at the second frame rate, so that it can be ensured that the displayed image is based on the second frame rate, and the definition of the displayed image is improved.
As a specific application example of this embodiment, the method is applied to an image synthesis system, where the image synthesis system includes a sending end and a receiving end, and as shown in fig. 7, the method includes:
for the sending end:
(1) acquiring a screen image, and 5 frames/second; collecting mouse information, 30 frames/second; extracting and synthesizing the mouse information into a screen image of 5 frames/second;
(2) and respectively sending the screen image of 5 frames/second and the mouse information of 30 frames/second to a receiving end.
For the receiving end with the synthesizing function:
(1) scratching the mouse image from the screen image;
(2) copy extended screen image 30 frames/second;
(3) synthesizing the screen image into mouse information of 30 frames/second;
(4) the composite screen image is displayed for 30 frames/second.
For a receiving end without a synthesizing function:
(1) discarding the mouse information for 30 frames/second;
(2) the screen image is displayed for 5 frames/second.
According to the image synthesis method provided by the embodiment, the mouse information is synthesized at the sending end, and whether the image needs to be the display image with the high frame rate is determined according to the requirement based on the synthesis function of the receiving end.
In this embodiment, an image synthesis apparatus is further provided, which is used to implement the above embodiments and preferred embodiments, and the description of the image synthesis apparatus is omitted. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
The present embodiment provides an image synthesizing apparatus, as shown in fig. 8, including:
a first obtaining module 81, configured to obtain a background image acquired at a first frame rate and foreground image information acquired at a second frame rate, where the first frame rate is smaller than the second frame rate;
an expansion module 82, configured to expand the background image acquired at the first frame rate based on the second frame rate to obtain a background image corresponding to the second frame rate;
a first synthesis module 83, configured to synthesize a background image corresponding to the second frame rate and foreground image information acquired at the second frame rate, and determine a first synthesized image;
a first sending module 84 for sending the first composite image.
The present embodiment provides an image synthesizing apparatus, as shown in fig. 9, comprising:
a second obtaining module 91, configured to obtain a background image acquired at a first frame rate and foreground image information acquired at a second frame rate, where the first frame rate is smaller than the second frame rate;
an extraction module 92, configured to perform image extraction on foreground image information acquired at the second frame rate based on the first frame rate to obtain foreground image information corresponding to the first frame rate;
a second synthesis module 93, configured to synthesize a background image acquired at the first frame rate and foreground image information corresponding to the first frame rate, and determine a second synthesized image;
a second sending module 94, configured to send the second composite image and the foreground image information acquired at the second frame rate.
The present embodiment also provides an image synthesizing apparatus, as shown in fig. 10, including:
a receiving module 101, configured to receive a second composite image determined according to the image synthesis method and foreground image information acquired at the second frame rate;
a third obtaining module 102, configured to obtain a local synthesizing function;
and the processing module 103 is configured to process the second composite image and the foreground image information acquired at the second frame rate based on the composite function, and determine a display image.
The image compositing apparatus in this embodiment is presented in the form of functional units, where a unit refers to an ASIC circuit, a processor and memory executing one or more software or fixed programs, and/or other devices that may provide the above-described functionality.
Further functional descriptions of the modules are the same as those of the corresponding embodiments, and are not repeated herein.
An embodiment of the present invention further provides an electronic device, which includes the image synthesizing apparatus shown in any one of fig. 8 to 10.
Referring to fig. 11, fig. 11 is a schematic structural diagram of an electronic device according to an alternative embodiment of the present invention, and as shown in fig. 11, the electronic device may include: at least one processor 201, such as a CPU (Central Processing Unit), at least one communication interface 203, memory 204, and at least one communication bus 202. Wherein a communication bus 202 is used to enable the connection communication between these components. The communication interface 203 may include a Display (Display) and a Keyboard (Keyboard), and the optional communication interface 203 may also include a standard wired interface and a standard wireless interface. The Memory 204 may be a high-speed RAM (Random Access Memory) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The memory 204 may optionally be at least one storage device located remotely from the processor 201. Wherein the processor 201 may be in connection with the apparatus as described in any of the fig. 8-10, the memory 204 stores an application program, and the processor 201 calls the program code stored in the memory 204 for performing any of the above-mentioned method steps.
The communication bus 202 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 202 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown in FIG. 11, but this is not intended to represent only one bus or type of bus.
The memory 204 may include a volatile memory (RAM), such as a random-access memory (RAM); the memory may also include a non-volatile memory (english: non-volatile memory), such as a flash memory (english: flash memory), a hard disk (english: hard disk drive, abbreviated: HDD) or a solid-state drive (english: SSD); memory 204 may also comprise a combination of the above types of memory.
The processor 201 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of a CPU and an NP.
The processor 201 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.
Optionally, memory 204 is also used to store program instructions. The processor 201 may call program instructions to implement an image synthesis method as shown in any of the embodiments of the present application.
Embodiments of the present invention further provide a non-transitory computer storage medium, where computer-executable instructions are stored, and the computer-executable instructions may execute the image synthesis method in any of the above method embodiments. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. An image synthesis method, comprising:
acquiring a background image acquired at a first frame rate and foreground image information acquired at a second frame rate, wherein the first frame rate is less than the second frame rate;
expanding the background image acquired at the first frame rate based on the second frame rate to obtain a background image corresponding to the second frame rate;
synthesizing the background image corresponding to the second frame rate and the foreground image information acquired at the second frame rate to determine a first synthesized image;
and sending the first composite image.
2. The method of claim 1, wherein the second frame rate is an integer multiple of the first frame rate, and the expanding the background image acquired at the first frame rate based on the second frame rate to obtain the background image corresponding to the second frame rate comprises:
determining an expansion coefficient by using the size relation between the second frame rate and the first frame rate;
and expanding the background image acquired at the first frame rate based on the expansion coefficient, and determining the background image corresponding to the second frame rate.
3. An image synthesis method, comprising:
acquiring a background image acquired at a first frame rate and foreground image information acquired at a second frame rate, wherein the first frame rate is less than the second frame rate;
performing image extraction on the foreground image information acquired at the second frame rate based on the first frame rate to obtain foreground image information corresponding to the first frame rate;
synthesizing the background image acquired at the first frame rate and foreground image information corresponding to the first frame rate to determine a second synthesized image;
and sending the second synthetic image and the foreground image information acquired at the second frame rate.
4. The method as claimed in claim 3, wherein the second frame rate is an integer multiple of the first frame rate, and the performing image extraction on the foreground image information collected at the second frame rate based on the first frame rate to obtain the foreground image information corresponding to the first frame rate comprises:
determining an extraction coefficient by using the size relation between the second frame rate and the first frame rate;
and extracting the foreground image information acquired at the second frame rate based on the extraction coefficient, and determining the foreground image information at the first frame rate.
5. An image synthesis method, comprising:
receiving a second synthetic image determined according to the image synthesis method of claim 3 or 4 and foreground image information of the second frame rate acquisition;
acquiring a local synthesis function;
and processing the second synthetic image and the foreground image information acquired at the second frame rate based on the synthetic function to determine a display image.
6. The method of claim 5, wherein when a synthesis function is locally available, the processing the second synthesized image and the foreground image information acquired at the second frame rate based on the synthesis function to determine a display image comprises:
matting out a foreground image in the second synthetic image based on the foreground image information to obtain a target background image corresponding to the first frame rate;
expanding the target background image corresponding to the first frame rate based on the second frame rate to obtain a target background image corresponding to the second frame rate;
and synthesizing the target background image corresponding to the second frame rate and the foreground image information acquired at the second frame rate to determine the display image.
7. The method as claimed in claim 6, wherein said matting a foreground image in the second composite image based on the foreground image information to obtain a target background image corresponding to the first frame rate comprises:
based on the middle position information of the foreground image information, matting and removing the foreground image from the corresponding position of the second synthetic image to obtain a background image to be processed;
acquiring an original background image covered by the foreground image in the background image acquired at the first frame rate;
and synthesizing the original background image and the corresponding background image to be processed, and determining a target background image corresponding to the first frame rate.
8. The method of claim 5, wherein when the local device does not have a synthesis function, the processing the second synthesized image and the foreground image information acquired at the second frame rate based on the synthesis function to determine a display image comprises:
and discarding the foreground image information acquired at the second frame rate, and determining the second composite image as the display image.
9. An electronic device, comprising:
a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the image composition method of any one of claims 1-2, or any one of claims 3-4, or any one of claims 5-8.
10. A computer-readable storage medium storing computer instructions for causing a computer to perform the image synthesis method of any one of claims 1-2, or any one of claims 3-4, or any one of claims 5-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111551043.5A CN114170122A (en) | 2021-12-17 | 2021-12-17 | Image synthesis method, image synthesis device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111551043.5A CN114170122A (en) | 2021-12-17 | 2021-12-17 | Image synthesis method, image synthesis device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114170122A true CN114170122A (en) | 2022-03-11 |
Family
ID=80487212
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111551043.5A Pending CN114170122A (en) | 2021-12-17 | 2021-12-17 | Image synthesis method, image synthesis device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114170122A (en) |
-
2021
- 2021-12-17 CN CN202111551043.5A patent/CN114170122A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8401316B2 (en) | Method and apparatus for block-based compression of light-field images | |
| KR101813196B1 (en) | Method, device, program, and recording medium for video communication | |
| WO2017193576A1 (en) | Video resolution adaptation method and apparatus, and virtual reality terminal | |
| US20140267583A1 (en) | Augmented Video Calls on Mobile Devices | |
| CN110868625A (en) | Video playing method and device, electronic equipment and storage medium | |
| CN108881743B (en) | Video image processing method, device and system based on splicing wall | |
| CN110569013B (en) | Image display method and device based on display screen | |
| CN110290398B (en) | Video issuing method and device, storage medium and electronic equipment | |
| WO2023193521A1 (en) | Video inpainting method, related apparatus, device and storage medium | |
| CN110913118B (en) | Video processing method, device and storage medium | |
| KR101551915B1 (en) | Device and method for video compression | |
| JP2019149785A (en) | Video conversion device and program | |
| CN114170122A (en) | Image synthesis method, image synthesis device, electronic equipment and storage medium | |
| CN114245027B (en) | Video data hybrid processing method, system, electronic equipment and storage medium | |
| CN107491934B (en) | 3D interview system based on virtual reality | |
| CN113810755B (en) | Panoramic video preview method and device, electronic equipment and storage medium | |
| US20080007648A1 (en) | Real time scaling and rotating decoded image data | |
| CN108933945B (en) | GIF picture compression method, device and storage medium | |
| CN106210590B (en) | A kind of image-pickup method, device and storage medium | |
| CN113873327B (en) | Video stream presentation method, device, equipment and storage medium | |
| WO2023207516A1 (en) | Live streaming video processing method and apparatus, electronic device, and storage medium | |
| CN115299033B (en) | Imaging device and imaging processing method | |
| CN111699672B (en) | Video control device and video control method | |
| CN112911186B (en) | Image storage method and device, electronic equipment and storage medium | |
| TWI493502B (en) | Processing method for image rotating and apparatus thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |