WO2022068067A1

WO2022068067A1 - Video conference information processing method and apparatus, electronic device and system

Info

Publication number: WO2022068067A1
Application number: PCT/CN2020/135587
Authority: WO
Inventors: 陈志明
Original assignee: 常熟九城智能科技有限公司
Priority date: 2020-09-30
Filing date: 2020-12-11
Publication date: 2022-04-07
Also published as: CN113271429A

Abstract

The present invention relates to the technical field of image processing, and provides a video conference information processing method and apparatus, an electronic device and a system. The video conference information processing method comprises: acquiring video conference information; determining a multi-terminal display image and the size information of a conference video stream displayed on the multi-terminal display image; de-noising the conference video stream displayed on the multi-terminal display image; and superposing the de-noised multi-terminal display image and a user interaction window, and then sending same to a video conference host end for display. Therefore, a video having high image quality and displaying interactive content is obtained, so that a user can also see interaction information while watching the video having high image quality.

Description

A video conference information processing method, device, electronic device and system

technical field

The present disclosure relates to the technical field of video conferences, and in particular, to a video conference information processing method, apparatus, electronic device, and system.

Background technique

With the development of network communication technology, users' demands for real-time video interaction are getting higher and higher, so a large number of video conferencing technologies and related devices have also been derived. Currently, there are a large number of video conferencing devices on the market, all of which have certain interactive functions (such as commenting, allowing/denying access to a conference, etc.).

technical problem

With the promotion of 5G and 4K technologies, video conferencing APPs will provide a better experience on various types of display terminals. However, currently on the market, the interactive functions in display terminals are usually independent of the displayed video conference content. Users need to Jumping out or interacting with the video conference through other channels cannot send or receive interactive instructions in real time during the video conference, which greatly reduces the user experience.

technical solutions

In order to at least overcome the above deficiencies in the prior art, the purpose of the present disclosure is to provide a video conference information processing method, device, electronic device and system.

In a first aspect, the present disclosure provides a method for processing video conference information. The method is executed by a video conference participant terminal, and is characterized by comprising:

acquiring video conference information, where the video conference information includes a conference video stream and a user command set, and the user command set includes commands used when the user participates in the video conference and interacts;

determining the multi-terminal display image and the size information of the conference video stream displayed on the multi-terminal display image;

performing denoising processing on the conference video stream displayed on the multi-terminal display image;

The denoised multi-terminal display image and the user interaction window are superimposed, and then sent to the video conference host for display, and the user interaction window is used to display the interactive content corresponding to each command in the user command set , the user interaction window includes a covered first area determined according to the size information.

In a second aspect, an embodiment of the present disclosure further provides a video conference information processing apparatus, which is characterized by comprising:

an acquisition unit, configured to acquire video conference information, where the video conference information includes a conference video stream and a user command set, and the user command set includes commands used when the user participates in the video conference and interacts;

a determining unit, configured to determine the multi-terminal display image and the size information of the conference video stream displayed on the multi-terminal display image;

a processing unit, configured to perform denoising processing on the conference video stream displayed on the multi-terminal display image; and

In a third aspect, an embodiment of the present disclosure further provides an electronic device, characterized in that it includes:

a transceiver to obtain video conference information, where the video conference information includes a conference video stream and a user command set, and the user command set includes commands used when the user participates in the video conference and interacts;

a first chip, for determining the multi-terminal display image and the size information of the conference video stream displayed on the multi-terminal display image;

a second chip, for performing denoising processing on the conference video stream displayed on the multi-terminal display image;

In a fourth aspect, an embodiment of the present disclosure further provides a video conference information processing system, the system includes a cloud server and the above video conference information processing apparatus, the cloud server includes a processor, a machine-readable storage medium, and a network interface, The machine-readable storage medium, the network interface, and the processor are connected through a bus system, the network interface is used to communicate with at least one video conference participant, and the machine-readable storage medium is used to store A program, instruction or code, the processor is configured to execute the program, instruction or code in the machine-readable storage medium to execute the video conference information processing method in the first aspect or any possible design of the first aspect.

In a fifth aspect, embodiments of the present disclosure provide a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when executed, cause a computer to execute the first aspect or any possible design of the first aspect The video conference information processing method in .

beneficial effect

Based on any one of the above aspects, the present disclosure obtains video conference information; determines the size information of the multi-terminal display image and the conference video stream displayed on the multi-terminal display image; performs denoising processing on the conference video stream displayed on the multi-terminal display image; The denoised multi-terminal display image and the user interaction window are superimposed, and then sent to the video conference host for display. As a result, a video with high image quality and interactive content is obtained, so that users can see interactive information while watching the video with high image quality, which enhances the user experience of the video conference.

Description of drawings

1 is a schematic diagram of an application scenario of a video conference information processing system provided by an embodiment of the present disclosure;

2 is a schematic flowchart of a method for processing video conference information provided by an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of functional modules of a video conference information processing apparatus provided by an embodiment of the present disclosure;

FIG. 4 is a schematic structural block diagram of a cloud server for implementing the above-mentioned video conference information processing method according to an embodiment of the present disclosure.

Embodiments of the present invention

The present disclosure will be specifically described below with reference to the accompanying drawings in the specification, and the specific operation methods in the method embodiments may also be applied to the apparatus embodiments or the system embodiments.

FIG. 1 is an interactive schematic diagram of a video conference information processing system 10 provided by an embodiment of the present disclosure. The video conference information processing system 10 may include a cloud server 100 and a video conference participant terminal 200 communicatively connected to the cloud server 100 . The video conference information processing system 10 shown in FIG. 1 is only a feasible example, and in other feasible embodiments, the video conference information processing system 10 may only include a part of the components shown in FIG. 1 or may also Include other components.

In this embodiment, the video conference participant terminal 200 may include a mobile device, a tablet computer, a laptop computer, etc., or any combination thereof. In some embodiments, mobile devices may include smart home devices, wearable devices, smart mobile devices, virtual reality devices, or augmented reality devices, etc., or any combination thereof. In some embodiments, the smart home devices may include control devices of smart electrical devices, smart monitoring devices, smart TVs, smart cameras, etc., or any combination thereof. In some embodiments, the wearable device may include a smart bracelet, smart shoelaces, smart glass, smart helmets, smart watches, smart clothing, smart backpacks, smart accessories, etc., or any combination thereof. In some embodiments, a smart mobile device may include a smartphone, a personal digital assistant, a gaming device, etc., or any combination thereof. In some embodiments, the virtual reality device and/or augmented reality device may include a virtual reality helmet, virtual reality glass, virtual reality patch, augmented reality helmet, augmented reality glass, or augmented reality patch, etc., or any combination thereof. For example, virtual reality devices and/or augmented reality devices may include various virtual reality products and the like.

In this embodiment, the IoT cloud server 100 and the video conference participant terminal 200 in the video conference information processing system 10 can cooperate to execute the video conference information processing method described in the following method embodiments, specifically the cloud server 100 and the video conference information processing method. For the part of the execution steps of the video conference participant terminal 200, reference may be made to the detailed description of the following method embodiments.

In order to solve the technical problems in the foregoing background technology, FIG. 2 is a schematic flowchart of a video conference information processing method provided by an embodiment of the present disclosure. The video conference information processing method provided by the present embodiment can be shown in FIG. 1 The cloud server 100 shown in FIG. 1 executes the video conference information processing method, and the video conference information processing method will be described in detail below.

Step S110, acquiring video conference information, where the video conference information includes a conference video stream and a user command set, and the user command set includes commands used when the user participates in the video conference and interacts;

Step S120, determining the multi-terminal display image and the size information of the conference video stream displayed on the multi-terminal display image;

Step S130, performing denoising processing on the conference video stream displayed on the multi-terminal display image;

Step S140, superimpose the denoised multi-terminal display image and the user interaction window, and then send it to the video conference host for display, and the user interaction window is used to display the commands corresponding to each command in the user command set. The interactive content, the user interaction window includes a covered first area determined according to the size information.

In a possible embodiment, step S120 further includes:

Step S121: Obtain the multi-terminal display image through the multi-terminal display image fusion technology.

In a possible embodiment, step S130 further includes:

A mark is added to the multi-terminal display image, and the mark is used to determine that the data to be denoised is the user source corresponding to the conference video stream displayed on the multi-terminal display image.

In a possible embodiment, step S120 further includes:

Step S122: According to the size of the video image in the conference video stream and the size of the multi-terminal display image, determine the size of the video image in the conference video stream displayed on the multi-terminal display image;

Step S123: Record the four-corner coordinate information or the four-side margin information of the video image in the conference video stream on the multi-terminal display image.

FIG. 3 is a schematic diagram of functional modules of the video conference information processing apparatus 300 according to an embodiment of the present disclosure. In this embodiment, the video conference information processing apparatus 300 may be divided into functional modules according to the method embodiments performed by the cloud server 100, that is, The following functional modules corresponding to the video conference information processing apparatus 300 may be used to execute various method embodiments executed by the cloud server 100 above. The video conference information processing apparatus 300 may include an acquisition unit 310, a determination unit 320, and a processing unit 330. The functions of each functional module of the video conference information processing apparatus 300 are described in detail below.

The obtaining unit 310 may be configured to perform the above-mentioned step S110, that is, to obtain video conference information, where the video conference information includes a conference video stream and a user command set, and the user command set includes commands used when the user participates in the video conference to interact. .

The determining unit 320 may be configured to perform the above-mentioned step S120, that is, to determine the size information of the multi-terminal display image and the conference video stream displayed on the multi-terminal display image.

The processing unit 330 may be configured to perform the above step S130, that is, to perform denoising processing on the conference video stream displayed on the multi-terminal display image; and denoising the multi-terminal display image and the user interaction window after the denoising processing It is superimposed, and then sent to the video conference host for display. The user interaction window is used to display the interactive content corresponding to each command in the user command set, and the user interaction window includes the size information determined. The first area to be covered.

It should be noted that it should be understood that the division of each module of the above apparatus is only a division of logical functions, and may be fully or partially integrated into a physical entity in actual implementation, or may be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also all be implemented in hardware; some modules can also be implemented in the form of calling software through processing elements, and some modules can be implemented in hardware. For example, the acquisition unit 310 may be a separately established processing element, or may be integrated into a certain chip of the above-mentioned device to be implemented, in addition, it may also be stored in the memory of the above-mentioned device in the form of program code, and processed by one of the above-mentioned devices The element invokes and executes the functions of the acquisition unit 310 above. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, and can also be implemented independently. The processing element described herein may be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above-mentioned method or each of the above-mentioned modules can be completed by an integrated logic circuit of hardware in the processor element or an instruction in the form of software.

For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more application specific integrated circuits (ASIC), or one or more digital microprocessors (digital) signal processor, DSP), or, one or more field programmable gate arrays (field programmable gate array, FPGA) and so on. For another example, when one of the above modules is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can invoke program codes. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

FIG. 4 shows a schematic diagram of the hardware structure of a cloud server 100 for implementing the above-mentioned control device provided by an embodiment of the present disclosure. As shown in FIG. 4 , the cloud server 100 may include a processor 110 , a machine-readable storage medium 120 , and a bus 130 and transceiver 140.

In a specific implementation process, at least one processor 110 executes computer-executable instructions stored in the machine-readable storage medium 120 (for example, included in the video conference information processing apparatus 300 shown in FIG. 3 ), so that the processor 110 can execute the above In the video conference information processing method according to the method embodiment, the processor 110, the machine-readable storage medium 120 and the transceiver 140 are connected through the bus 130, and the processor 110 can be used to control the transceiver 140 to send and receive actions, thereby Data transmission and reception can be performed with the aforementioned video conference participant terminal 200 .

For the specific implementation process of the processor 110, reference may be made to the various method embodiments executed by the cloud server 100 above, and the implementation principles and technical effects thereof are similar, and are not described again in this embodiment.

In the above-mentioned embodiment shown in FIG. 4 , it should be understood that the processor may be a central processing unit (English: Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (English: Digital Signal Processor) , DSP), application specific integrated circuit (English: Application Specific Integrated Circuit, ASIC) etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the invention can be directly embodied as executed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

Machine-readable storage medium 120 may include high-speed RAM memory, and may also include non-volatile storage NVM, such as at least one disk storage.

The bus 130 may be an industry standard architecture (Industry Standard Architecture, ISA) bus, peripheral device interconnect (Peripheral Component Interconnect, PCI) bus or Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus, etc. The bus 130 may be classified into an address bus, a data bus, a control bus, and the like. For convenience of representation, the buses in the drawings of the present application are not limited to only one bus or one type of bus.

In addition, an embodiment of the present disclosure also provides a readable storage medium, where computer-executable instructions are stored in the readable storage medium, and when a processor executes the computer-executable instructions, the above video conference information processing method is implemented. .

The above-mentioned readable storage medium may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM) , Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic Disk or Optical Disk. A readable storage medium can be any available medium that can be accessed by a general purpose or special purpose computer.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, but not to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions described in the foregoing embodiments can still be modified, or some or all of the technical features thereof can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present disclosure. scope.

Claims

A method for processing video conference information, the method being executed by a video conference participant terminal, characterized in that it includes:

acquiring video conference information, where the video conference information includes a conference video stream and a user command set, and the user command set includes commands used when the user participates in the video conference and interacts;

determining the multi-terminal display image and the size information of the conference video stream displayed on the multi-terminal display image;

performing denoising processing on the conference video stream displayed on the multi-terminal display image;

The denoised multi-terminal display image and the user interaction window are superimposed, and then sent to the video conference host for display, and the user interaction window is used to display the interactive content corresponding to each command in the user command set , the user interaction window includes a covered first area determined according to the size information.
The method according to claim 1, wherein the determining the multi-terminal display image comprises:

The multi-terminal display image is obtained through the multi-terminal display image fusion technology.
The method according to claim 2, wherein the method further comprises:

A mark is added to the multi-terminal display image, and the mark is used to determine that the data to be denoised is the user source corresponding to the conference video stream displayed on the multi-terminal display image.
The method according to claim 1, wherein the determining the size information of the conference video stream displayed on the multi-terminal display image comprises:

According to the size of the video image in the conference video stream and the size of the multi-terminal display image, determine the size of the video image in the conference video stream displayed on the multi-terminal display image;

Record the four-corner coordinate information or four-side margin information of the video image in the conference video stream on the multi-terminal display image.
A video conference information processing device, characterized in that it includes:

an acquisition unit, configured to acquire video conference information, where the video conference information includes a conference video stream and a user command set, and the user command set includes commands used when the user participates in the video conference and interacts;

a determining unit, configured to determine the multi-terminal display image and the size information of the conference video stream displayed on the multi-terminal display image;

a processing unit, configured to perform denoising processing on the conference video stream displayed on the multi-terminal display image; and

The denoised multi-terminal display image and the user interaction window are superimposed, and then sent to the video conference host for display, and the user interaction window is used to display the interactive content corresponding to each command in the user command set , the user interaction window includes a covered first area determined according to the size information.
An electronic device, comprising:

a transceiver to obtain video conference information, where the video conference information includes a conference video stream and a user command set, and the user command set includes commands used when the user participates in the video conference and interacts;

a first chip, for determining the multi-terminal display image and the size information of the conference video stream displayed on the multi-terminal display image;

a second chip, for performing denoising processing on the conference video stream displayed on the multi-terminal display image;

The denoised multi-terminal display image and the user interaction window are superimposed, and then sent to the video conference host for display, and the user interaction window is used to display the interactive content corresponding to each command in the user command set , the user interaction window includes a covered first area determined according to the size information.
A video conference information processing system, the system comprising a cloud server and the video conference information processing device according to claim 5, wherein the cloud server comprises a processor, a machine-readable storage medium and a network interface, the The machine-readable storage medium, the network interface and the processor are connected through a bus system, the network interface is used to communicate with at least one video conference participant, and the machine-readable storage medium is used to store programs , instruction or code, the processor is configured to execute the program, instruction or code in the machine-readable storage medium, so as to execute the video conference information processing method of any one of claims 1-4.
A computer-readable storage medium for storing instructions/executable codes, when the instructions/executable codes are executed by a processor of an electronic device, the electronic device is made to implement any one of claims 1-4. method described.