CN108289185B

CN108289185B - Video communication method, device and terminal equipment

Info

Publication number: CN108289185B
Application number: CN201710014525.4A
Authority: CN
Inventors: 李凯
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2017-01-09
Filing date: 2017-01-09
Publication date: 2021-08-13
Anticipated expiration: 2037-01-09
Also published as: CN108289185A

Abstract

The embodiment of the invention provides a video communication method, a video communication device and terminal equipment, wherein the method comprises the following steps: the method comprises the steps that a first terminal device establishes video communication connection with a second terminal device and obtains a video image collected by an image collecting device of the first terminal device; determining the effect of the selected filter; determining a data throughput type of the filter effect; if the data processing capacity type accords with a first type, adding the filter effect to the video image in a CPU of the first terminal device; if the data processing capacity type is in accordance with a second type, indicating the GPU of the first terminal equipment to add the filter effect on the video image; determining a video image added with a video special effect at least according to the video image added with the filter effect; and transmitting the video image added with the video special effect to the second terminal equipment. The implementation of the invention reduces the use limitation of video communication based on video special effects.

Description

Video communication method, device and terminal equipment

Technical Field

The invention relates to the technical field of communication, in particular to a video communication method, a video communication device and terminal equipment.

Background

Video communication refers to a mode of communication between users by transmitting video images, and after video communication connection is established between terminal devices such as mobile phones of users, video images acquired by cameras can be mutually transmitted through a network, so that video communication is realized.

In order to meet the personalized requirements of video communication, the current video communication mode can also support the addition of video special effects in video images, such as the addition of images of face pendants such as beards and animal ears for faces in the video images, and the addition of filter effects such as cartoons and cartoons for the video images.

A GPU (graphics processing unit), which is a conventional device for processing graphics, generally implements adding video effects to video images on the GPU; however, the GPU needs to be responsible for the display of a graphical interface, the image presentation, and the like in the terminal device, and with the increase of graphics processing tasks, the terminal device needs to set the GPU with sufficiently strong performance to meet the requirement of adding a video special effect on a video image, which provides a higher requirement for the performance configuration of the terminal device, resulting in that the use limit of video communication based on the video special effect is increasingly large.

Disclosure of Invention

In view of this, embodiments of the present invention provide a video communication method, an apparatus, and a terminal device, so as to reduce the use limitation of video communication based on video special effects.

In order to achieve the above purpose, the embodiments of the present invention provide the following technical solutions:

a video communication method is applied to a first terminal device, and comprises the following steps:

establishing video communication connection with second terminal equipment, and acquiring a video image acquired by an image acquisition device of the first terminal equipment;

determining the effect of the selected filter;

determining a data throughput type of the filter effect;

if the data processing capacity type accords with a first type, adding the filter effect to the video image in a CPU of the first terminal device;

if the data processing capacity type is in accordance with a second type, indicating the GPU of the first terminal equipment to add the filter effect on the video image; the data processing capacity corresponding to the first type corresponds to a data processing capacity range set for the CPU, and the data processing capacity corresponding to the first type is lower than that corresponding to the second type;

determining a video image added with a video special effect at least according to the video image added with the filter effect;

and transmitting the video image added with the video special effect to the second terminal equipment.

An embodiment of the present invention further provides a video communication apparatus, which is applied to a first terminal device, and the apparatus includes:

the connection establishing module is used for establishing video communication connection with the second terminal equipment;

the video image acquisition module is used for acquiring a video image acquired by the image acquisition device of the first terminal equipment;

the filter effect determining module is used for determining the selected filter effect;

the type determining module is used for determining the data processing amount type of the filter effect;

a first filter effect adding module, configured to add the filter effect to the video image in a CPU of the first terminal device if the data throughput type matches a first type;

a second filter effect adding module, configured to instruct the GPU of the first terminal device to add the filter effect to the video image if the data throughput type matches a second type; the data processing capacity corresponding to the first type corresponds to a data processing capacity range set for the CPU, and the data processing capacity corresponding to the first type is lower than that corresponding to the second type;

the video image determination module for adding the special effect is used for determining the video image for adding the video special effect at least according to the video image for adding the filter effect;

and the video image transmission module is used for transmitting the video image added with the video special effect to the second terminal equipment.

An embodiment of the present invention further provides a terminal device, including:

the CPU is used for establishing video communication connection with second terminal equipment and acquiring a video image acquired by an image acquisition device of the first terminal equipment; determining the effect of the selected filter; determining a data throughput type of the filter effect; if the data processing capacity type accords with a first type, adding the filter effect to the video image in a CPU of the first terminal device; if the data processing capacity type is in accordance with a second type, indicating the GPU of the first terminal equipment to add the filter effect on the video image; the data processing capacity corresponding to the first type corresponds to a data processing capacity range set for the CPU, and the data processing capacity corresponding to the first type is lower than that corresponding to the second type; determining a video image added with a video special effect at least according to the video image added with the filter effect; transmitting a video image added with a video special effect to the second terminal equipment;

and the GPU is used for adding the filter effect on the video image when the data processing quantity type is in accordance with a second type under the instruction of the CPU.

Based on the above technical solution, in the video communication method provided in the embodiment of the present invention, after selecting the filter effect, the first terminal device may determine whether the filter effect is implemented in the CPU or the GPU according to the data throughput type of the filter effect; the processing complexity corresponding to the data processing amount type is low, when the processing complexity is in accordance with a first type, the filter effect can be added to the video image in the CPU of the first terminal device, when the processing complexity corresponding to the data processing amount type is high, and when the processing complexity is in accordance with a second type, the GPU of the first terminal device can be instructed to add the filter effect to the video image; therefore, the embodiment of the invention can reasonably distribute the processing devices for executing the filter effect addition according to the data processing amount type of the filter effect, so that the data processing pressure related to the filter effect addition can be shared between the CPU and the GPU, and the terminal equipment has certain comprehensive performance, thereby meeting the requirement of adding the video special effect on the video image without strengthening the performance configuration of a certain aspect such as the GPU, providing possibility for the low-configuration terminal equipment to realize the filter effect and reducing the use limitation of video communication based on the video special effect.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

Fig. 1 is a block diagram of a video communication system according to an embodiment of the present invention;

fig. 2 is a signaling flow chart of a video communication method according to an embodiment of the present invention;

fig. 3 is another signaling flow diagram of a video communication method according to an embodiment of the present invention;

FIG. 4 is a flow chart of a method of determining a selected video effect;

fig. 5 is a block diagram of a terminal device;

FIG. 6 is a schematic view of a process for implementing the addition of a face pendant to a video image;

FIG. 7 is another schematic flow diagram for implementing the addition of face hangings and filter effects to a video image;

FIG. 8 is a flow chart of a method of video encoding a video image with added video effects;

FIG. 9 is a schematic diagram of the performance of skin beauty skincare, skin makeup, tested on the GPU of iPhone 5S;

FIG. 10 is a schematic diagram of the performance of skin beauty skincare, skin makeup, tested on the GPU of iPhone 4S;

FIG. 11 is a schematic diagram of testing the performance of a face recognition technique on different iPhone handsets;

fig. 12 is a block diagram of a video communication apparatus according to an embodiment of the present invention;

fig. 13 is another block diagram of a video communication apparatus according to an embodiment of the present invention;

fig. 14 is a block diagram of another structure of a video communication apparatus according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Fig. 1 is a block diagram of a video communication system according to an embodiment of the present invention, and referring to fig. 1, the video communication system may include: a first terminal device 10 and a second terminal device 20;

the first terminal device and the second terminal device may be user-side devices having image acquisition devices such as cameras and data processing capabilities, such as smart phones, tablet computers, and notebook computers with cameras.

When a first user (a user of a first terminal device) and a second user (a user of a second terminal device) perform video communication, video images can be mutually transmitted through the first terminal device and the second terminal device to realize video communication, and data interaction between the first terminal device and the second terminal device can be realized through a network, specifically, the data interaction can be realized through a network server providing video communication service, such as an IM (instant messaging) server having a video communication function.

Fig. 1 is a schematic diagram of a double-person video communication scenario, showing that the embodiment of the present invention can also support multi-person video communication, such as supporting video communication of group users.

In the embodiment of the invention, the processes of processing and transmitting the video images acquired by the image acquisition device by the first terminal equipment and the second terminal equipment, receiving the video images and displaying the video images are the same; the video communication process provided by the embodiment of the invention is described below by taking a first terminal device as a video image sending device and a second terminal device as a video image receiving device as an example; obviously, the second terminal device may also become a video image transmitting device when transmitting a video image to the first terminal device, and the first terminal device may also become a video image receiving device when receiving a video image transmitted by the second terminal device; the video communication flow of this process is the same as the principle of the communication flow in which the first terminal device is a video image transmitting device and the second terminal device is a video image receiving device, and can be referred to each other.

Fig. 2 shows a signaling flow chart of a video communication method provided by an embodiment of the present invention, and referring to fig. 2, the flow chart may include:

step S10, the first terminal device establishes a video communication connection with the second terminal device.

Optionally, the first terminal device may establish a video communication connection with the second terminal device through a network server supporting video communication; a network server supporting video communication, such as an IM (Instant Messaging) server with a video communication function.

Step S11, the first terminal device obtains a video image captured by the image capturing device of the first terminal device.

After the first terminal equipment and the second terminal equipment establish video communication connection, the first terminal equipment and the second terminal equipment can mutually transmit video images; in the embodiment of the present invention, the first terminal device may add a video special effect to a video image acquired by an image acquisition device such as a camera, and transmit the video image added with the video special effect to the second terminal device; the same processing is carried out on the second terminal equipment; therefore, the first terminal device needs to acquire the video image acquired by the image acquisition device, and the video image acquired by the image acquisition device is transmitted to the CPU of the first terminal device.

Step S12, the first terminal device determines the selected filter effect.

The filter effect is like the filter effect of a cartoon special effect, the filter effect of different color special effects and the like.

And step S13, the first terminal equipment determines the data processing amount type of the filter effect.

And step S14, if the data processing amount type is in accordance with a first type, adding the filter effect for the video image in the CPU of the first terminal equipment.

And step S15, if the data processing amount type is in accordance with a second type, instructing the GPU of the first terminal device to add the filter effect on the video image.

The data processing amount corresponding to the first type corresponds to a data processing amount range set for the CPU, and the data processing amount corresponding to the first type is lower than the data processing amount corresponding to the second type.

In the embodiment of the invention, in order to realize reliable communication of the video image added with the video special effect, when the filter effect is added to the video image, the CPU or the GPU of the first terminal equipment is selected to realize the addition of the filter effect according to the processing complexity of the filter effect to be added;

specifically, when the processing complexity of the filter effect is low, the filter effect is added to the video image on the CPU of the first terminal device, and when the processing complexity of the filter effect is high, the filter effect is added to the video image on the GPU of the first terminal device.

It should be noted that although the GPU is a dedicated Graphics processing device, once the GPU is used for image processing, an OpenGL interface (an Open Graphics Library, a professional Graphics program interface with a cross-programming language and a cross-platform programming interface specification) generally needs to be called for realization, and the calling of the OpenGL interface will greatly occupy processing resources of the GPU, resulting in an increase in the occupation of processing resources of the terminal device, and once the GPU is currently involved in more Graphics processing tasks, the GPU with sufficiently strong performance needs to be set to meet the requirement of adding a video special effect on a video image, which will put forward a higher requirement on the performance configuration of the terminal device; therefore, according to the processing complexity of the filter effect, when the processing complexity of the filter effect is low and the filter effect can be realized by the CPU, the addition of the filter effect can be realized on the CPU of the first terminal device, the calling of an OpenGL interface is reduced, and the processing pressure of a GPU is reduced; when the processing complexity of the filter effect is high and the filter effect cannot be realized by the CPU, the filter effect is realized in the GPU, and the smooth realization of the filter effect addition is ensured;

therefore, the terminal equipment has certain comprehensive performance, the requirement of adding the video special effect on the video image can be met, performance configuration of a certain aspect such as GPU (graphics processing unit) does not need to be enhanced, the possibility that the low-configuration terminal equipment can achieve the filter effect is provided, and the use limitation of video communication based on the video special effect is reduced.

Optionally, the processing complexity of the filter effect may be determined by the data throughput of the filter effect, generally speaking, the lower the data throughput of the filter effect is, the lower the processing complexity is, the higher the data throughput of the filter effect is, the higher the processing complexity is;

according to the embodiment of the invention, the data processing type of the filter effect can be defined according to the data processing amount of the filter effect, and a first type and a second type are defined; the data processing capacity corresponding to the first type corresponds to a data processing capacity range set for the CPU, and the data processing capacity corresponding to the first type is lower than that corresponding to the second type;

the threshold values of the data processing amounts corresponding to the first type and the second type may be threshold values with low processing complexity and high processing complexity, which are analyzed in advance according to the CPU data processing capability of the first terminal device; and for the CPUs with different data processing capabilities of different models, the determined limit values of the data processing amounts corresponding to the first type and the second type may be different.

When the data processing amount type of the selected filter effect is determined to be in accordance with the first type, the filter effect can be determined to be realized by a CPU, and the filter effect is added to the video image in the CPU of the first terminal device; and when the data processing amount type of the selected filter effect is in accordance with the second type, determining that the filter effect is not suitable for being realized by the CPU, and realizing adding the filter effect on the video image by the GPU for professional graphic processing.

Optionally, the type of the data throughput of the selected filter effect may be determined according to the data throughput of the filter effect, if the data throughput of the filter effect corresponds to the data throughput corresponding to the first type, it may be determined that the type of the data throughput corresponds to the first type, and if the data throughput of the filter effect corresponds to the data throughput corresponding to the second type, it may be determined that the type of the data throughput corresponds to the second type;

optionally, the filter effect data may be locally stored in the first terminal device, or downloaded from a network, and each filter effect may carry a corresponding data throughput type identifier, where the data throughput type identifier may correspond to a data throughput of a filter effect, which is analyzed in advance based on the data throughput of the filter effect.

Optionally, the data processing amount range set for the CPU may be determined according to a model capability of the first terminal device, where the model capability may be an upper limit value of the data processing amount of the CPU (generally determined by the number of cores of the CPU, etc.), and in the embodiment of the present invention, the upper limit value of the data processing amount of the CPU may be determined, and a CPU occupation ratio range of the CPU processing filter effect may be set; determining the data processing capacity range by the data processing capacity upper limit value and the CPU occupation proportion range (such as multiplication of the data processing capacity upper limit value and the CPU occupation proportion range);

obviously, this is only an optional way of the data processing amount range set by the CPU, and the set data processing amount range should be within the data processing capacity range of the CPU, and the embodiment of the present invention may also determine the current data processing amount range of the CPU (for example, the current idle resource with the set resource ratio) in combination with the current idle resource of the CPU, so as to implement dynamic change of the data processing amount range set by the CPU according to the idle resource of the CPU.

In the embodiment of the present invention, the data processing amount ranges set for the CPU are different in different models, that is, the data processing amounts corresponding to the first type and the second type are different in different models.

Optionally, the filter effect realized in the CPU may be based on data in YUV format, that is, if the filter effect data and the video image acquired from the image acquisition device are not data in YUV format, the filter effect may be added to the video image only by converting the filter effect data into YUV format in the embodiment of the present invention;

optionally, the filter effect implemented in the GPU may be based on data in RGB format; the embodiment of the invention can render the video image into the GPU and store the video image into an RGB format, after the GPU adds the filter effect on the video image, the video image with the filter effect is transmitted to the CPU, and then the CPU is further converted into a YUV format so as to facilitate subsequent video coding processing and the like.

And step S16, determining a video image with the video special effect at least according to the video image with the filter effect.

Optionally, the video image added with the video special effect may be a video image added with a filter effect and a face pendant image, in addition to the video image added with the filter effect.

And step S17, transmitting the video image with the video special effect to the second terminal equipment.

Optionally, the first terminal device may perform video coding processing on the video image with the video special effect added thereto, and then transmit the video image after the coding processing to the second terminal device; optionally, before performing video coding processing on the video image with the video special effect, the first terminal device may further perform coding pre-processing on the video image with the video special effect, where the coding pre-processing includes at least one of the following processing manners: and noise reduction and sharpening, such as buffing and the like.

Optionally, when the filter effect is added to the video image in the CPU of the first terminal device, the embodiment of the present invention may call a predetermined filter effect implementation algorithm to add the filter effect to the video image;

when the GPU of the first terminal device is instructed to add the filter effect to the video image, the embodiment of the present invention may invoke an OpenGL interface to instruct the GPU to implement an algorithm with a predetermined filter effect through the OpenGL interface, and add the filter effect to the video image.

In the video communication method provided by the embodiment of the invention, after the first terminal device selects the filter effect, whether the filter effect is realized in the CPU or the GPU can be determined according to the data processing amount type of the filter effect; the processing complexity corresponding to the data processing amount type is low, when the processing complexity is in accordance with a first type, the filter effect can be added to the video image in the CPU of the first terminal device, when the processing complexity corresponding to the data processing amount type is high, and when the processing complexity is in accordance with a second type, the GPU of the first terminal device can be instructed to add the filter effect to the video image; therefore, the embodiment of the invention can reasonably distribute the processing devices for executing the filter effect addition according to the data processing amount type of the filter effect, so that the data processing pressure related to the filter effect addition can be shared between the CPU and the GPU, and the terminal equipment has certain comprehensive performance, thereby meeting the requirement of adding the video special effect on the video image without strengthening the performance configuration of a certain aspect such as the GPU, providing possibility for the low-configuration terminal equipment to realize the filter effect and reducing the use limitation of video communication based on the video special effect.

Optionally, if a face pendant needs to be added to the video image, the first terminal device may adjust the frame rate of the video image subjected to the video special effect processing according to the current network bandwidth, and based on the principle that the higher the current network bandwidth is, the higher the frame rate is, the lower the current network bandwidth is, and the lower the frame rate is, under the condition that the network bandwidth is higher, the frame rate of the video image subjected to the video special effect processing is increased, and on the basis of ensuring the reliability of video communication, the quality of the video image is increased; and under the condition of lower network bandwidth, the frame rate of the video image subjected to the video special effect processing is reduced, and the reliability of video communication is guaranteed.

Optionally, the adjusting of the frame rate of the video image subjected to the video special effect processing may be implemented by adjusting the frame rate of the video image acquired by the image acquisition device, specifically as shown in fig. 3, where fig. 3 is another flowchart of the video communication method, and the process may include:

step S20, the first terminal device obtains the current network bandwidth.

The way of acquiring the current network bandwidth by the first terminal device may be various, for example, the first terminal device interacts with a server having a network bandwidth detection function to perform bandwidth detection packet, so as to detect the current network bandwidth.

Step S21, the first terminal device determines, according to the first corresponding relationship between the preset network bandwidth range and the image acquisition frame rate, the image acquisition frame rate corresponding to the network bandwidth range where the current network bandwidth is located.

The network bandwidth range is positively correlated with the corresponding image acquisition frame rate, that is, the higher the network bandwidth range is, the higher the corresponding image acquisition frame rate is.

The embodiment of the invention can preset different network bandwidth ranges and corresponding image acquisition frame rates, wherein the image acquisition frame rates corresponding to the different network bandwidth ranges are different, and the higher the network bandwidth range is, the higher the corresponding image acquisition frame rate is (namely, the network bandwidth range is positively correlated with the corresponding image acquisition frame rate), so as to form a first corresponding relation.

Optionally, the frame rate of the video image acquired by the first terminal device is limited to the acquisition frame rate of an image acquisition device such as a camera of the first terminal device; therefore, the upper limit value of the image acquisition frame rate can be the acquisition frame rate of the image acquisition device; table 1 below shows an alternative schematic of the network bandwidth range and the image acquisition frame rate, which can be referred to, wherein the maximum frame rate is the acquisition frame rate of the image acquisition apparatus.

Image acquisition frame rate	Network bandwidth range
		fps<8	Bandwidth of<80kbps
8<fps<＝10	Bandwidth of<100kbps
		10<fps<＝12	100kbps<Bandwidth of<150kbps
12<fps<＝15	150kbps<Bandwidth of<200kbps
		fps>15	Bandwidth of>200kbps

TABLE 1 schematic Table of first correspondence

After determining the current network bandwidth of the first terminal device, the embodiment of the present invention may determine the image acquisition frame rate corresponding to the network bandwidth range in which the current network bandwidth of the first terminal device is located in the first corresponding relationship.

Step S22, the first terminal device acquires a video image acquired by the image acquisition device of the first terminal device according to the determined image acquisition frame rate.

The image acquisition frame rate is in positive correlation with the current network bandwidth of the first terminal device, namely the higher the current network bandwidth of the first terminal device is, the higher the image acquisition frame rate is, the lower the current network bandwidth is, and the lower the image acquisition frame rate is;

correspondingly, the video image acquired by the image acquisition frame rate acquisition device can acquire more video image frame numbers in unit time under the condition that the current network bandwidth of the first terminal device is higher, and then the video image frame numbers in the unit time are used for carrying out video special effect addition, so that the image quality of the video image added with the video special effect is higher;

under the condition that the current network bandwidth of the first terminal device is lower, the embodiment of the invention can acquire the video image acquired by the image acquisition device at a lower image acquisition frame rate, so that when the current network bandwidth is lower, a smaller number of video image frames is acquired in unit time, and then video special effect addition is carried out by using the number of the video image frames in the unit time, thereby ensuring the reliability of video communication.

Alternatively, steps S20 to S22 may be regarded as an alternative implementation manner of acquiring the video image captured by the image capturing device of the first terminal apparatus in step S11 shown in fig. 2.

And step S23, the first terminal equipment adds the selected video special effect to the acquired video image.

Adding a selected video special effect to the video image, such as adding a face pendant to the video image, and/or adding a filter effect to the video image; if the filter effect is selected, determining whether to realize the filter effect in the CPU or the GPU according to the method shown in FIG. 2;

if the face pendant is added, the face pendant can be added on the GPU;

optionally, further, if a filter effect is implemented on the GPU, the GPU may add the filter effect on the video image to which the face pendant is added, or add the face pendant on the video image to which the filter effect is added; if the filter effect is realized on the CPU, the CPU can add the filter effect on the video image which is fed back by the GPU and added with the face pendant.

And step S24, the first terminal device transmits the video image with the video special effect to the second terminal device.

In the embodiment of the invention, the first terminal device can adjust the image acquisition frame rate according to the current network bandwidth, and the current network bandwidth and the adjusted image acquisition frame rate are in a positive correlation relationship; the first terminal equipment can obtain the adjusted image obtaining frame rate, obtain the video image collected by the image collecting device of the first terminal equipment, and add the selected video special effect to the obtained video image, so that the frame rate of the video image subjected to the video special effect processing is in positive correlation with the current network bandwidth; the current network bandwidth is high, so that the frame rate of the video image subjected to video special effect processing can be improved on the basis of ensuring the communication reliability, and the image quality of the video image added with the video special effect is improved; when the current network bandwidth is low, the frame rate of the video image subjected to the video special effect processing can be reduced, the resource consumption of the terminal equipment and the network bandwidth used for video communication are reduced, and the reliability of the video communication is guaranteed.

According to the video communication method provided by the embodiment of the invention, the frame rate of the video image subjected to video special effect processing can be adapted to the current network bandwidth by dynamically adjusting the image acquisition frame rate corresponding to the current network bandwidth, so that the video communication added with the video special effect can be reliably carried out, and the reliability of the video communication is ensured.

Further, the embodiment of the present invention may determine the video special effect selected by the user, and when the video special effect selected by the user corresponds to the device configuration information of the first terminal device and the current network bandwidth,

adding a video special effect selected by a user to the obtained video image; the method and the device have the advantages that under the conditions that the device configuration is low and the network bandwidth is low, the number of executable video special effect types is reduced, and the resource consumption of the terminal device is further reduced;

optionally, fig. 4 is a flowchart illustrating a method for determining a selected video special effect according to an embodiment of the present invention, where the method is applicable to a first terminal device, and referring to fig. 4, the method may include:

step S100, determining at least one currently executable video special effect type according to the device configuration information of the first terminal device and the current network bandwidth.

Optionally, in the embodiment of the present invention, a second corresponding relationship between the device configuration level, the network bandwidth range, and the video special effect type may be preset; the device configuration level and the network bandwidth range are higher, the number of the corresponding video special effect types is larger, and one video special effect type can correspond to at least one video special effect;

the higher the device configuration level and the network bandwidth range are, the higher the device resource and the bandwidth resource can be provided by the first terminal device, and more video special effect types can be realized; table 2 below shows a second correspondence between the device configuration level, the network bandwidth range, and the video special effect type, which can be referred to; the data processing capacity of the basic filter is low, and the data processing capacity of the complex filter is high;

TABLE 2

The embodiment of the invention can obtain the equipment configuration information of the memory, the CPU, the GPU and the like of the first terminal equipment, and determine the equipment configuration grade of the first terminal equipment based on the preset equipment configuration grade grading strategy, generally speaking, the higher the equipment configuration grade is, the higher the equipment configuration of the terminal equipment is, for example, the larger the memory is, the more the number of CPU cores is and the like;

after determining the device configuration level of the first terminal device and the network bandwidth range where the current network bandwidth is located, the embodiment of the present invention may determine the device configuration level of the device configuration information of the first terminal device and the network bandwidth range where the current network bandwidth is located in the second corresponding relationship and determine at least one video special effect type currently executable by the first terminal device.

And step S110, displaying the video special effect corresponding to the executable at least one video special effect type.

Optionally, the first terminal device may display the video special effect selection area on the video communication interface, where one displayed video special effect type may correspond to at least one video special effect, so that the user selects a video special effect that needs to be added to the video image.

And step S120, determining the video special effect selected from the displayed video special effects.

Optionally, the embodiment of the present invention may determine the filter effect selected from the displayed video special effect, that is, by using the method shown in fig. 4, step S12 shown in fig. 2 is implemented;

optionally, in addition to the selection of the filter effect, the embodiment of the invention can also realize the selection of the face pendant image.

With reference to the methods shown in fig. 3 and fig. 4, in the embodiment of the present invention, when the current network bandwidth of the first terminal device is higher, the video image to be added with the special effect is obtained at a higher image obtaining frame rate, and when the device configuration level of the first terminal device is higher, the video special effect is added to the video image by using more executable video special effect types, so that on the basis of ensuring the reliability of video communication, the image quality of the video image to which the video special effect is added is improved; when the current bandwidth of the first terminal device is low and the device configuration level is low, the embodiment of the invention can obtain the video image to be added with the special effect at a low image obtaining frame rate, and add the video special effect to the video image by using fewer executable video special effect types, thereby reducing the resource consumption and the network bandwidth consumption of the terminal device in the video communication process and ensuring the reliability of the video communication.

Alternatively, the video communication method described above may be executed by a CPU of the first terminal device, fig. 5 shows a structure of the first terminal device, and referring to fig. 5, the first terminal device may include: an image acquisition device 11, a CPU (Central Processing Unit) 12, and a GPU (Graphics Processing Unit) 13; the structure of the second terminal device is similar to that of the first terminal device;

optionally, after the video image acquired by the image acquisition device is acquired at the determined image acquisition frame rate, the embodiment of the invention can realize the addition of the face pendant on the video image; in order to further reduce the CPU resource consumption of the first terminal device, the embodiment of the invention can use the grayscale image channel image in the video image to realize the determination of the adding position of the face pendant in the video image;

optionally, fig. 6 shows a schematic flow of the first terminal device implementing adding of the face pendant to the video image, where the flow may include:

in step S30, the CPU acquires the video image acquired by the image acquisition device at the determined image acquisition frame rate.

When a first user performs video communication with a second user, a first terminal device can start an image acquisition device such as a camera and the like to acquire a local video image, wherein the video image generally comprises a human face of the first user and a background image of an environment background where the first user is located; after determining the corresponding image acquisition frame rate according to the current network bandwidth, the first terminal device can acquire the video image from the image acquisition device at the corresponding frame rate.

Alternatively, the video image captured by the image capturing device may be a YUV (where Y refers to luminance and UV is collectively referred to as chrominance) video image, or may be an RGB (where R refers to red, G refers to green, and B refers to blue) video image, and the format of the captured video image may be determined according to the type of the image capturing device.

Step S31, the CPU determines the selected face pendant image.

When a user of a first terminal device needs to add a face pendant special effect on a face of a video image, the user can locally select a face pendant image from a network server or the first terminal device, and a CPU (central processing unit) can determine the selected face pendant image after detecting a face pendant image selection instruction input by the user.

Optionally, the video communication interface of the first terminal device may display a face pendant image selection area, the face pendant image selection area may display a face pendant image downloaded from a network server and/or locally stored in the first terminal device, and the user may select the face pendant image from the face pendant image selection area.

In step S32, the CPU extracts a grayscale image channel image of the acquired video image.

Based on the face pendant image selected in step S21, the CPU may determine that the face pendant image needs to be subsequently added to the video image, and in order to identify the addition position of the face pendant image in the video image, the CPU needs to extract a grayscale image channel image of the video image for each frame of the acquired video image.

For the purposes of reducing data processing amount and reducing CPU resource consumption, the CPU can position the human face characteristic points only according to the grayscale image channel images in the video images; specifically, the CPU may extract a grayscale image in the video image, such as extracting a Y-channel image in the video image in YUV format (the Y-channel is a luminance channel, and the image representation of the Y-channel is similar to the representation of the grayscale image), or extracting a G-channel image in the video image in RGB format (the image representation of the G-channel is similar to the representation of the grayscale image).

In step S33, the CPU identifies the positions of the face feature points in the grayscale channel image.

Optionally, the CPU may process the grayscale image channel image through a face recognition technique, so as to position the face feature points in the grayscale image channel image, where the face feature points are feature points of five sense organs (eyebrows, eyes, ears, noses, mouths) of a face, and the like; specifically, the CPU may determine a face region in the grayscale image channel image by a face detection technique, perform face feature point positioning (e.g., feature point positioning of five sense organs) on the face region, and determine the position of each face feature point in the grayscale image channel image; the position of each obtained face characteristic point in the grayscale image channel image can be regarded as the position of each face characteristic point in the video image acquired by the image acquisition device;

optionally, because an angle difference (generally, an angle difference of 90 degrees or 180 degrees) may exist between the image acquired by the image acquisition device and the image finally displayed on the screen, a face in the video image acquired by the image acquisition device may not be upright.

Furthermore, because the size of the video image acquired by the camera is generally larger, the embodiment of the invention can directly position the human face feature point for the grayscale image channel image, and can also perform reduction processing on the grayscale image channel image to obtain the grayscale image channel reduced image, realize the positioning of the human face feature point based on the grayscale image channel reduced image, and then convert the position of the human face feature point positioned in the grayscale image channel reduced image into the position of the human face feature point in the grayscale image channel image.

Alternatively, the position of the face feature point in the image may be defined by coordinates of the face feature point in the image.

Step S34, the CPU determines the adding position of the face hanger image in the video image according to the face feature point corresponding to the face hanger image and the position of each face feature point.

Optionally, each face pendant image has a corresponding face feature point, and the face pendant image is added to the corresponding face feature point, for example, the face pendant image of a rabbit ear corresponds to an ear feature point of a face, and the face pendant image of glasses corresponds to a glasses feature point of the face; after determining the face hanger image to be added to the video image, the CPU may determine the adding position of the face hanger image in the video image according to the face feature point corresponding to the face hanger image and the position of each face feature point in the grayscale image channel image determined in step S33;

specifically, according to the face feature points corresponding to the selected face pendant image, the embodiment of the present invention may match the positions corresponding to the face feature points corresponding to the face pendant image from the positions of the face feature points in the grayscale image channel image, so as to obtain the adding positions of the face pendant image in the video image.

Step S35, the CPU renders the video image and the face pendant image into a GPU of the first terminal device, so that the video image and the face pendant image in a first image format are generated in the GPU; and transmitting the adding position to the GPU.

Optionally, the CPU may invoke an OpenGL interface (an Open Graphics Library, a professional graphical program interface of a cross-programming language, cross-platform programming interface specification), render the video image and the face suspension image to a texture of the GPU, and store the video image and the face suspension image in the first image format in the GPU;

optionally, the first image format may be an RGB format; correspondingly, if the video image and/or the face pendant image are in YUV format, the CPU needs to perform image format conversion when rendering the video image and the face pendant image to the texture of the GPU, so as to store the video image in RGB format and the face pendant image in the GPU;

when rendering the video image and the face suspension image to the texture of the GPU, the CPU may further notify the GPU of the addition position determined in step S34, so that the GPU implements addition of the face suspension image on the video image.

And step S36, adding the video image in the first image format and the face pendant image by the GPU according to the adding position to obtain a special effect video image in the first image format.

Optionally, the GPU adds the video image in the first image format and the face pendant image, or adds the face pendant image to the video image; specifically, the GPU may add the face pendant image to the video image in the first image format according to the addition position to obtain the special-effect video image in the first image format.

Optionally, if it is determined that the filter effect is added in the GPU based on the method shown in fig. 2, the GPU may further add the filter effect to the video image to obtain a special-effect video image in the first image format, to which the filter effect and the face suspension image are added.

Step S37, the GPU transmits the special effect video image in the first image format to the CPU.

Optionally, the CPU may convert the special effect video image in the first image format into a second image format, perform video coding processing on the special effect video image in the second image format, and transmit the special effect video image after the video coding processing to the second terminal device; optionally, the second image format may be a YUV format;

optionally, for convenience of video coding processing, if the special effect video image in the first image format is in an RGB format, the CPU may perform conversion from an RBG format to a YUV format to obtain a special effect video image in a YUV format, and perform video coding processing on the special effect video image in the YUV format to obtain a special effect video image after the video coding processing;

further, if the filter effect is determined to be achieved on the CPU, the CPU may add the filter effect based on the video image with the face pendant added in the second image format; therefore, the CPU can perform video coding processing on the video image with the second image format and the added face pendant and filter effect, for example, perform coding preprocessing on the video image with the YUV format, wherein the coding preprocessing comprises at least one of the following processing modes: noise reduction sharpening processing (typically only processing Y-channel images), such as buffing processing (typically only processing Y-channel images), some very special filters (typically only processing UV-channel images), etc.; and further transmitting the video image after the coding processing to the second terminal equipment.

The following describes a flow of adding a face pendant and a filter effect to a video image provided by an embodiment of the present invention, taking a video image acquired by an image acquisition device as a YUV format (an optional form of a second image format), and for convenience of description, the following description is only schematic from the perspective of processing by a first terminal device, and the flow may be as shown in fig. 7 and includes:

and step S40, the CPU acquires the YUV video image acquired by the camera at the determined image acquisition frame rate.

Step S41, the CPU determines the selected face suspension image and the selected filter effect, and determines that the data throughput type of the filter effect is the first type.

Step S42, the CPU extracts a Y-channel image in the YUV video image.

Alternatively, the Y channel image is only an alternative form of the grayscale image channel image.

The image expression of the Y channel in the YUV video image is similar to the expression of the gray-scale image channel, and the position of the subsequent face characteristic point is positioned by extracting the Y channel image in the YUV video image, so that the data processing pressure of the first terminal equipment can be reduced, and the resource consumption of a CPU (Central processing Unit) is reduced.

In step S43, the CPU reduces the Y-channel image according to the set scale to obtain a reduced Y-channel image.

Alternatively, the Y-channel reduced image is only an alternative form of the grayscale image channel reduced image.

Optionally, the size of the reduced image may be set, and under the condition that the size of the video image acquired by the camera is fixed, the reduction ratio of the video image may be set, so that the video image is reduced according to the set reduction ratio, the data processing amount related to the face recognition technology is reduced, and the face recognition processing efficiency is improved; optionally, the size of the reduced image does not affect the accuracy of face recognition, and the specific size may be set according to actual conditions.

And reducing the Y-channel image to obtain a reduced Y-channel image.

In step S44, the CPU rotates the Y-channel reduced image according to the set rotation angle.

The set rotation angle may be a difference angle between a video image acquired by the camera and a screen display image, such as 90 degrees or 180 degrees, and the reduced Y-channel image is rotated by the set rotation angle, so that a human face in the reduced Y-channel image is upright.

In step S45, the CPU recognizes the position of each face feature point in the rotated Y-channel reduced image.

Optionally, the embodiment of the present invention may apply a face recognition technology to perform face recognition processing on the rotated Y-channel image, and locate the position of each face feature point in the rotated Y-channel image.

Step S46, the CPU converts the position of each face feature point in the rotated Y-channel reduced image into the position of each face feature point in the Y-channel image, to obtain the position of each face feature point in the YUV video image.

Optionally, the specific conversion process may be: determining the positions of the face feature points in the Y-channel reduced image that has been rotated in the reverse direction (opposite to the direction in which the Y-channel reduced image was rotated in step S35) after the Y-channel reduced image has been rotated in the reverse direction by the set rotation angle; and determining the position of each face characteristic point in the enlarged Y-channel image (namely the Y-channel image) after the reversely-rotated Y-channel reduced image is enlarged according to the set proportion.

The position of each face feature point in the Y-channel image obtained after conversion can be regarded as the position of each face feature point in the YUV video image.

And step S47, the CPU determines the adding position of the face pendant image in the YUV video image according to the face characteristic point corresponding to the face pendant image and the position of each face characteristic point in the YUV video image.

Optionally, a is set as a pixel point on the video image, and the size of an area covered by the face position of the face pendant on the video image is set as a point B, and the point C at the adding position can be determined through the points a and B in the embodiment of the present invention; specifically, C ═ alpha a + (1-alpha) × B, and alpha is a transparency value of the face suspension image.

Step S48, calling an OpenGL interface to render the YUV video image and the face pendant image into a GPU, and storing the video image in an RGB format and the face pendant image in the GPU; and transmitting the adding position to the GPU.

And step S49, adding the video image in the RGB format and the face pendant image by the GPU according to the adding position to obtain the special effect video image in the RGB format.

Because the obtained video image is a multi-frame video image sequence, the embodiment of the invention can realize that the face pendant image is added on the multi-frame video image sequence by correspondingly superposing the face pendant image and the video image sequence frame by frame.

And step S50, the GPU transmits the special effect video image in the RGB format to the CPU.

Step S51, the CPU converts the RGB format special effect video image into YUV format special effect video image, and adds filter effect.

Optionally, if the CPU determines that the data throughput type of the filter effect is the second type, the CPU may further instruct the GPU to add the filter effect to the video image, so that the GPU may transmit the special-effect video image, to which the face suspension image and the filter effect are added, in the RGB format to the CPU, and convert the special-effect video image into the YUV format.

Optionally, the CPU may further perform encoding preprocessing on the YUV-format special-effect video image with the added face image and the filter effect, perform video encoding processing on the special-effect video image after the encoding preprocessing, and transmit the special-effect video image after the video encoding processing to the second terminal device;

optionally, the CPU may transmit the special effect video image after the video coding processing to the second terminal device through the communication module of the first terminal device; the communication module is a communication device with network communication capability, such as a WIFI (wireless fidelity) communication module and a GPRS (general packet radio service) communication module.

Optionally, the video image captured by the image capturing device may also be in RGB format, and the processing manner in this case is different from that shown in fig. 6 and 7 in that: the CPU can acquire RGB video images acquired by the camera according to the determined image acquisition frame rate, and position location of each face characteristic point in the RGB video images is carried out based on the G channel images; meanwhile, the CPU can directly call an OpenGL interface to render the RGB video image and the selected RGB face pendant image into a GPU; subsequently, after the CPU obtains the RGB-format special effect video image transmitted by the GPU, the CPU may convert the RGB-format special effect video image into a YUV-format special effect video image to perform video encoding preprocessing and video encoding postprocessing, and further transmit the video image after the video encoding processing to the second terminal device.

Optionally, in the embodiment of the present invention, the implementation of the filter effect on the CPU may be based on a video image in a YUV format and implemented by filter effect data; the implementation of the filter effect on the GPU may be based on the RGB format video image, and the filter effect data implementation.

Optionally, on one hand, the CPU may render the video image acquired from the image acquisition device and the face pendant image selected by the user into the GPU, and instruct the GPU to add the selected face pendant image and the filter effect to the video image; optionally, the GPU may add a filter effect to the video image, and then add the face pendant image to the video image to which the filter effect is added, or add the face pendant image to the video image, and then add the filter effect to the video image to which the face pendant image is added;

optionally, on the other hand, the CPU may render the video image acquired from the image acquisition device and the face suspension image selected by the user into the GPU, and instruct the GPU to add the selected face suspension image to the video image; therefore, the CPU can add the selected filter effect to the obtained video image which is transmitted by the GPU and is added with the face pendant image.

Optionally, when video coding processing is performed on a video image added with a video special effect, the embodiment of the present invention may further determine a coding resolution during video coding according to the device configuration level and the network bandwidth range;

alternatively, fig. 8 shows a flowchart of a method for performing video coding processing on a video image to which a video special effect is added, where the method is applicable to a CPU of a first terminal device, and referring to fig. 8, the flowchart may include:

step S300, determining the device configuration level of the device configuration information of the first terminal device and the coding resolution corresponding to the network bandwidth range in which the current network bandwidth is located according to a preset third corresponding relationship between the device configuration level, the network bandwidth range and the coding resolution.

Optionally, the coding resolution may be in positive correlation with both the device configuration level and the network bandwidth range, and the upper limit value of the coding resolution is selected as the highest resolution of the terminal device; that is, if the network bandwidth is high and the device configuration level is high, the embodiment of the present invention may limit the highest resolution of the terminal device, for example, iOS is 640x480, the Android face pendant is 480x360, and the Android filter is 1280x720, and when the face pendant and the filter are used simultaneously, the limit is 480x 360; table 3 below shows an alternative illustration of the third correspondence, which can be referred to.

Device configuration level	Network bandwidth range	Coding resolution
			First class (Low end type)	Bandwidth of<80kbps	192x144
First class (Low end type)	Bandwidth of<100kbps	320x240
			Second grade (middle-end type)	100kbps<Bandwidth of<150kbps	320x240
Second grade (middle-end type)	150kbps<Bandwidth of<200kbps	480x36
			Third grade (high-end type)	Bandwidth of>200kbps	480x360

TABLE 3

Step S310, performing video encoding processing on the video image to which the video special effect is added at the determined encoding resolution.

Therefore, the higher the network bandwidth of the terminal equipment is and the higher the equipment configuration level is, the higher the coding resolution can be adopted for video coding processing, and the image quality of the video image added with the video special effect is improved; and the lower the network bandwidth is, the lower the equipment configuration level is, the video coding processing can be performed by adopting the relatively lower coding resolution, the occupation of CPU and network resources is reduced, and the reliability of video communication is guaranteed.

Optionally, after the second terminal device obtains the video image added with the video special effect and transmitted by the first terminal device, the second terminal device may perform video decoding processing on the video image and display the video image.

It can be seen that the adding of the video special effect is executed at the first terminal device which transmits the video image, and the second terminal device which receives the video image can directly decode the video image added with the video special effect, so as to realize the display of the video image, achieve the effect of the video image added with the video special effect browsed by the first terminal device, namely the video image added with the video special effect sent by the first terminal device, and improve the synchronization of the video communication between the first terminal device and the second terminal device;

it is worth mentioning that even if the first terminal device adds a video special effect on the video image and transmits the video special effect to the second terminal device, the real-time transmission of the video image is not influenced, and the synchronization of the video communication is greatly improved; through research and test of the inventor of the invention, as shown in fig. 9, fig. 10 and fig. 11, on the basis of video images acquired by a traditional transmission camera, video special effect processing such as various face processing technologies, virtual props, various color filters and the like is added, and the requirements of real-time video call can be met; fig. 9 is a schematic diagram of testing the performance of skin beautifying and skin polishing on the GPU of iPhone5S, fig. 10 is a schematic diagram of testing the performance of skin beautifying and skin polishing on the GPU of iPhone4S, and fig. 11 is a schematic diagram of testing the performance of face recognition technology (face detection, tracking, feature point positioning of five sense organs) on different iPhone mobile phones.

The following describes a video communication apparatus provided by an embodiment of the present invention, and the video communication apparatus described below may be referred to in correspondence with the video communication method described above.

Fig. 12 is a block diagram of a video communication apparatus according to an embodiment of the present invention, where the apparatus is applicable to a CPU of a first terminal device, and referring to fig. 12, the apparatus may include:

a connection establishing module 100, configured to establish a video communication connection with a second terminal device;

a video image obtaining module 200, configured to obtain a video image collected by an image collecting device of the first terminal device;

a filter effect determination module 300 configured to determine a selected filter effect;

a type determining module 400, configured to determine a data throughput type of the filter effect;

a first filter effect adding module 500, configured to add the filter effect to the video image in a CPU of the first terminal device if the data throughput type matches a first type;

a second filter effect adding module 600, configured to instruct the GPU of the first terminal device to add the filter effect to the video image if the data throughput type is in accordance with a second type; the data processing capacity corresponding to the first type corresponds to a data processing capacity range set for the CPU, and the data processing capacity corresponding to the first type is lower than that corresponding to the second type;

a video image with a special effect determination module 700, configured to determine a video image with a special video effect according to the video image with a filter effect;

a video image transmission module 800, configured to transmit the video image with the video special effect added thereto to the second terminal device.

Optionally, as shown in fig. 12, the apparatus may further include:

a data processing amount range determining module 900, configured to determine an upper limit of the data processing amount of the CPU, and set a CPU occupation ratio range of the CPU processing filter effect; and determining the data processing capacity range according to the data processing capacity upper limit value and the CPU occupation proportion range.

Optionally, the first filter effect adding module 500 is configured to add the filter effect to the video image in the CPU of the first terminal device, and specifically includes:

and calling a preset filter effect implementation algorithm to add the filter effect to the video image.

Optionally, the second filter effect adding module 600 is configured to instruct the GPU of the first terminal device to add the filter effect to the video image, and specifically includes:

and indicating the GPU to realize an algorithm with a preset filter effect through an OpenGL interface, and adding the filter effect to the video image.

Optionally, the video image obtaining module 200 is configured to obtain a video image collected by an image collecting device of the first terminal device, and specifically includes:

acquiring the current network bandwidth;

determining an image acquisition frame rate corresponding to a network bandwidth range where the current network bandwidth is located according to a first corresponding relation between a preset network bandwidth range and the image acquisition frame rate; wherein, the network bandwidth range is positively correlated with the corresponding image acquisition frame rate;

and acquiring the video image acquired by the image acquisition device of the first terminal equipment according to the determined image acquisition frame rate.

Optionally, fig. 13 shows another structural block diagram of the video communication apparatus according to the embodiment of the present invention, and in combination with fig. 12 and 13, the apparatus may further include:

a display module 1000, configured to determine at least one currently executable video special effect type according to the device configuration information of the first terminal device and a current network bandwidth; displaying a video special effect corresponding to the executable at least one video special effect type; wherein one video effect type corresponds to at least one video effect.

Correspondingly, the filter effect determining module 300 is configured to determine the selected filter effect, and specifically includes:

a filter effect selected from the displayed video effects is determined.

Optionally, the presentation module 1000 is configured to determine at least one currently executable video special effect type according to the device configuration information of the first terminal device and the current network bandwidth, and specifically includes:

calling a second corresponding relation between a preset equipment configuration level, a network bandwidth range and a video special effect type; the higher the equipment configuration level and the network bandwidth range are, the more the number of the corresponding video special effect types is;

and determining the video special effect type corresponding to the equipment configuration level of the equipment configuration information of the first terminal equipment, the network bandwidth range where the current network bandwidth is located and the equipment configuration level of the equipment configuration information of the first terminal equipment in the second corresponding relation.

Optionally, fig. 14 shows another block diagram of a video communication apparatus according to an embodiment of the present invention, and in combination with fig. 13 and fig. 14, the apparatus may further include:

a face pendant image selection module 1100 for determining a face pendant image selected from the displayed video effects;

an image acquisition module 1200 for adding a face pendant, configured to extract a grayscale image channel image of the acquired video image; identifying the position of each face characteristic point in the gray-scale image channel image; determining the adding position of the face pendant image in the video image according to the face characteristic points corresponding to the face pendant image and the positions of the face characteristic points; rendering the video image and the face pendant image into a GPU of the first terminal device to generate the video image and the face pendant image in a first image format in the GPU; and transmitting the adding position to the GPU; and acquiring a special effect video image in a first image format transmitted by the GPU, wherein the special effect video image is a video image added with a face pendant image at the adding position.

Optionally, the video image determination module 700 for adding a special effect is configured to determine, according to at least the video image with the filter effect, a video image with the video special effect, and specifically includes:

determining a video image added with a face pendant image and a filter effect to obtain a video image added with a video special effect; wherein, the human face pendant image is added on the video image before or after the filter effect.

Optionally, the video image transmission module 800 is configured to transmit the video image with the video special effect to the second terminal device, and specifically includes:

determining a video image with a second image format and added with a video special effect;

carrying out video coding processing on the video image with the second image format and the added video special effect;

and transmitting the video image subjected to the video coding processing to second terminal equipment.

Optionally, the image obtaining module 1200 with a face pendant added thereto is configured to identify positions of each face feature point in the grayscale image channel image, and specifically includes:

reducing the grayscale image channel image according to a set proportion to obtain a grayscale image channel reduced image;

rotating the reduced image of the grayscale image channel according to a set rotation angle;

identifying the position of each human face characteristic point in the rotated grayscale image channel reduced image;

and converting the position of each human face characteristic point in the rotated grayscale image channel reduced image into the position of each human face characteristic point in the grayscale image channel image.

Optionally, the grayscale image channel image is a Y channel image of a YUV format video image, or a G channel image of an RGB format video image;

the first image format is an RGB format, and the second image format is a YUV format.

instructing the GPU of the first terminal device to add the selected face pendant image and the filter effect to the video image to obtain the video image added with the face pendant image and the filter effect;

or instructing the GPU to add the selected face pendant image to the video image so as to add the selected filter effect to the obtained video image added with the face pendant image to obtain the video image added with the face pendant image and the filter effect.

carrying out video coding processing on the video image added with the video special effect;

and transmitting the video image which is subjected to the video coding processing and added with the video special effect to the second terminal equipment.

Optionally, the video image transmission module 600 is configured to perform video coding processing on a video image added with a video special effect, and specifically includes:

determining the equipment configuration level of the equipment configuration information of the first terminal equipment and the coding resolution corresponding to the network bandwidth range of the current network bandwidth according to a preset third corresponding relation among the equipment configuration level, the network bandwidth range and the coding resolution;

and carrying out video coding processing on the video image added with the video special effect at the determined coding resolution.

Optionally, an embodiment of the present invention further provides a terminal device, where a structure of the terminal device may be as shown in fig. 5, and the terminal device includes: a CPU and a GPU;

in the stage of transmitting the video image, the CPU can be used for establishing video communication connection with a second terminal device and acquiring the video image acquired by the image acquisition device of the first terminal device; determining the effect of the selected filter; determining a data throughput type of the filter effect; if the data processing capacity type accords with a first type, adding the filter effect to the video image in a CPU of the first terminal device; if the data processing capacity type is in accordance with a second type, indicating the GPU of the first terminal equipment to add the filter effect on the video image; the data processing capacity corresponding to the first type corresponds to a data processing capacity range set for the CPU, and the data processing capacity corresponding to the first type is lower than that corresponding to the second type; determining a video image added with a video special effect at least according to the video image added with the filter effect; transmitting a video image added with a video special effect to the second terminal equipment;

the GPU may be configured, as instructed by the CPU, to add the filter effect on the video image when the data throughput type is of a second type.

Other functions of the CPU and GPU may be described with reference to the corresponding parts above.

The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video communication method applied to a first terminal device, the method comprising:

establishing video communication connection with a second terminal device, and acquiring the current network bandwidth;

acquiring a video image acquired by an image acquisition device of the first terminal equipment according to the determined image acquisition frame rate;

determining at least one currently executable video special effect type according to the equipment configuration information of the first terminal equipment and the current network bandwidth;

displaying a video special effect corresponding to the executable at least one video special effect type; wherein one video special effect type corresponds to at least one video special effect;

determining a filter effect selected from the displayed video effects;

determining a data throughput type of the filter effect;

if the data processing capacity type is in accordance with a second type, indicating the GPU of the first terminal equipment to add the filter effect on the video image; the data processing amount corresponding to the first type corresponds to a data processing amount range set for the CPU, the data processing amount corresponding to the first type is lower than the data processing amount corresponding to the second type, the limit values of the data processing amounts corresponding to the first type and the second type are analyzed in advance according to the CPU data processing capacity of the first terminal equipment, and the data processing amount range is determined by combining with the current idle resource of the CPU, so that the data processing amount range set for the CPU is dynamically changed along with the change of the current idle resource of the CPU;

2. The video communication method according to claim 1, wherein the determination process of the data processing amount range set by the CPU includes:

determining the upper limit value of the data processing capacity of the CPU, and setting the CPU occupation proportion range of the filter effect of the CPU;

and determining the data processing capacity range according to the data processing capacity upper limit value and the CPU occupation proportion range.

3. The video communication method according to claim 1, wherein said adding the filter effect to the video image in the CPU of the first terminal device comprises:

calling a preset filter effect implementation algorithm, and adding the filter effect to the video image;

the instructing the GPU of the first terminal device to add the filter effect on the video image comprises:

4. The video communication method according to claim 1, wherein the determining at least one currently executable video special effect type according to the device configuration information of the first terminal device and the current network bandwidth comprises:

5. The video communication method according to claim 1, wherein the method further comprises:

determining a face pendant image selected from the displayed video effects;

the method further comprises the following steps:

extracting a gray scale image channel image of the acquired video image;

identifying the position of each face characteristic point in the gray-scale image channel image;

determining the adding position of the face pendant image in the video image according to the face characteristic points corresponding to the face pendant image and the positions of the face characteristic points;

rendering the video image and the face pendant image into a GPU of the first terminal device to generate the video image and the face pendant image in a first image format in the GPU; and transmitting the adding position to the GPU;

and acquiring a special effect video image in a first image format transmitted by the GPU, wherein the special effect video image is a video image added with a face pendant image at the adding position.

6. The video communication method according to claim 5, wherein the determining a video image to which a video special effect is added based on at least the video image to which the filter effect is added comprises:

determining a video image added with a face pendant image and a filter effect to obtain a video image added with a video special effect; wherein, the human face pendant image is added on the video image before or after the filter effect;

the transmitting the video image added with the video special effect to the second terminal device comprises:

7. The video communication method of claim 5, wherein the identifying the location of each facial feature point in the grayscale channel image comprises:

8. The video communication method according to claim 6, wherein the grayscale image channel image is a Y-channel image of a YUV format video image, or a G-channel image of an RGB format video image;

9. The video communication method according to claim 1, wherein the determining a video image to which a video special effect is added based on at least the video image to which the filter effect is added comprises:

10. The video communication method according to claim 1, wherein said transmitting a video image to which a video special effect is added to the second terminal device comprises:

11. The video communication method according to claim 10, wherein said video encoding processing of the video image to which the video special effect is added comprises:

12. A video communication apparatus, applied to a first terminal device, the apparatus comprising:

a second filter effect adding module, configured to instruct the GPU of the first terminal device to add the filter effect to the video image if the data throughput type matches a second type; the data processing amount corresponding to the first type corresponds to a data processing amount range set for the CPU, the data processing amount corresponding to the first type is lower than the data processing amount corresponding to the second type, the limit values of the data processing amounts corresponding to the first type and the second type are analyzed in advance according to the CPU data processing capacity of the first terminal equipment, and the data processing amount range is determined by combining with the current idle resource of the CPU, so that the data processing amount range set for the CPU is dynamically changed along with the change of the current idle resource of the CPU;

the video image transmission module is used for transmitting the video image added with the video special effect to the second terminal equipment;

the video image acquisition module is configured to acquire a video image acquired by an image acquisition device of the first terminal device, and specifically includes: acquiring the current network bandwidth; determining an image acquisition frame rate corresponding to a network bandwidth range where the current network bandwidth is located according to a first corresponding relation between a preset network bandwidth range and the image acquisition frame rate; wherein, the network bandwidth range is positively correlated with the corresponding image acquisition frame rate; acquiring a video image acquired by an image acquisition device of the first terminal equipment according to the determined image acquisition frame rate;

the display module is used for determining at least one currently executable video special effect type according to the equipment configuration information of the first terminal equipment and the current network bandwidth; displaying a video special effect corresponding to the executable at least one video special effect type; wherein one video special effect type corresponds to at least one video special effect;

the filter effect determining module is configured to determine a selected filter effect, and specifically includes: a filter effect selected from the displayed video effects is determined.

13. The video communication apparatus according to claim 12, wherein the apparatus further comprises:

the data processing capacity range determining module is used for determining the upper limit value of the data processing capacity of the CPU and setting the CPU occupation proportion range of the filter processing effect of the CPU; determining the data processing capacity range with the CPU occupation ratio range according to the data processing capacity upper limit value;

the first filter effect adding module is configured to add the filter effect to the video image in the CPU of the first terminal device, and specifically includes:

the second filter effect adding module is configured to instruct the GPU of the first terminal device to add the filter effect to the video image, and specifically includes:

14. A terminal device is characterized in that the terminal device is a first terminal device; the first terminal device includes:

the CPU is used for establishing video communication connection with second terminal equipment and acquiring a video image acquired by an image acquisition device of the first terminal equipment and the current network bandwidth; determining an image acquisition frame rate corresponding to a network bandwidth range where the current network bandwidth is located according to a first corresponding relation between a preset network bandwidth range and the image acquisition frame rate; wherein, the network bandwidth range is positively correlated with the corresponding image acquisition frame rate; acquiring a video image acquired by an image acquisition device of the first terminal equipment according to the determined image acquisition frame rate; determining at least one currently executable video special effect type according to the equipment configuration information of the first terminal equipment and the current network bandwidth; displaying a video special effect corresponding to the executable at least one video special effect type; wherein one video special effect type corresponds to at least one video special effect; determining a filter effect selected from the displayed video effects; determining a data throughput type of the filter effect; if the data processing capacity type accords with a first type, adding the filter effect to the video image in a CPU of the first terminal device; if the data processing capacity type is in accordance with a second type, indicating the GPU of the first terminal equipment to add the filter effect on the video image; the data processing amount corresponding to the first type corresponds to a data processing amount range set for the CPU, and the data processing amount corresponding to the first type is lower than the data processing amount corresponding to the second type; determining a video image added with a video special effect at least according to the video image added with the filter effect; transmitting a video image added with a video special effect to the second terminal device, wherein the data processing capacity range is determined by combining the current idle resource of the CPU, so that the data processing capacity range set for the CPU is dynamically changed along with the change of the current idle resource of the CPU;

15. A computer-readable storage medium, characterized in that a program is stored in the computer-readable storage medium, which program, when being executed by a processor, is adapted to carry out the steps of the video communication method according to any one of claims 1 to 11.