TWI807504B - Method, device and storage medium for audio processing of virtual meeting room - Google Patents

Abstract

This application discloses a method, device and storage medium for audio processing of virtual meeting room, and relates to a technical field of virtual meeting room. The method includes: seting the number of mesh vertex according to a seat distribution of the virtual meeting room; obtaining a first voiceprint information of a speaker, the first voiceprint information includes the frequency, amplitude and phase difference of a speech signal; adjusting the frequency or amplitude of the first voiceprint information according to the number of mesh vertex to obtain a second voiceprint information; determining a seat of the speaker in the virtual meeting room according to the second voiceprint information. This application can simulate the sound source characteristics of the speaker and make the speaker's voice recognizable.

Description

Audio processing method, device and storage medium for virtual conference room

The present application relates to the technical field of virtual meeting rooms, in particular to an audio processing method, device and storage medium for virtual meeting rooms.

Virtual Meeting Room (Virtual Meeting Room, VMR) is an efficient and convenient network meeting room. Users can quickly and efficiently set up virtual meetings with other users through mobile terminal products such as mobile phones and computers, without being limited by time and space, and feel the effect of immersive meeting communication. The current virtual conference room enlarges the speaker's image, making it difficult to distinguish the voices of different speakers. When there are multiple speakers speaking at the same time in the virtual meeting room, it is difficult for the user to distinguish the speech content of each speaker.

This application provides an audio processing method, device and storage medium for a virtual meeting room, so as to improve the recognizability of the speaker's voice.

The first aspect of the present application provides an audio processing method for a virtual conference room, including: setting the number of grid vertices according to the seat distribution of the virtual conference room. The first voiceprint information of the speaker is extracted, and the first voiceprint information includes the frequency, amplitude and phase difference of the voice signal. The frequency or amplitude of the first voiceprint information is adjusted according to the number of grid vertices to obtain the second voiceprint information. The speaker's seat in the virtual conference room is determined according to the second voiceprint information.

In one embodiment, setting the number of grid vertices according to the seat distribution of the virtual conference room includes: setting different numbers of grid vertices in the area covered by each seat, so as to establish a corresponding relationship between seats and the number of grid vertices.

In another embodiment, adjusting the frequency or amplitude of the first voiceprint information according to the number of grid vertices includes: when the number of grid vertices in the area covered by the first seat is greater than the number of grid vertices in the area covered by the second seat, increasing the frequency of the first voiceprint information from the first seat, or lowering the frequency of the first voiceprint information from the second seat, so that the frequency of the first voiceprint information from the first seat is greater than the frequency of the first voiceprint information from the second seat.

In another embodiment, adjusting the frequency or amplitude of the first voiceprint information according to the number of grid vertices includes: when the number of grid vertices in the area covered by the first seat is greater than the number of grid vertices in the area covered by the second seat, increasing the amplitude of the first voiceprint information from the first seat, or reducing the amplitude of the first voiceprint information from the second seat, so that the amplitude of the first voiceprint information from the first seat is greater than the amplitude of the first voiceprint information from the second seat.

In another implementation manner, after the speaker's seat in the virtual meeting room is determined according to the second voiceprint information, the audio processing method further includes: capturing eye movement direction information of the meeting participant. Determine the concentration of the participants according to the eye movement direction information, and the value of the concentration is 0 or 1. Determine whether the participants are interested in the meeting topic according to the degree of concentration.

In another embodiment, determining the concentration of the participant according to the eye movement direction information includes: marking the concentration as 1 when the eye movement direction of the participant is facing the speaker. When the participant's eye movement direction is away from the speaker, mark the concentration as 0.

In another embodiment, the audio processing method further includes: when there are multiple speakers, counting the value of the concentration of the participants when each speaker speaks. Determine the concentration of the participants on the conference topic according to the value of the concentration.

In another implementation manner, determining whether a participant is interested in a conference topic according to the degree of concentration includes: determining that the participant is interested in the conference topic when the value of the concentration degree is greater than or equal to a preset interest threshold. When the value of the concentration is less than the interest threshold, it is determined that the participant is not interested in the meeting topic.

The second aspect of the present application provides an audio processing device, including a server, a master device, and a slave device. The master device is used to initiate a virtual meeting, the server is used to construct a virtual conference room according to instructions from the master device, and the slave device is used to enter the virtual conference room according to a connection from the master device. The server includes a first processor and a first memory. The first processor runs a computer program or code stored in the first memory to implement the audio processing method of the embodiment of the present application.

The third aspect of the present application provides a storage medium for storing computer programs or codes. When the computer programs or codes are executed by a processor, the audio processing method of the embodiment of the present application is realized.

In this embodiment of the present application, the number of grid vertices in the area covered by each seat in the virtual conference room is associated with the first voiceprint information, and the frequency or amplitude of the first voiceprint information from different seats is adjusted according to the number of grid vertices to obtain more recognizable second voiceprint information, thereby establishing a correspondence between each seat and the second voiceprint information. In this way, the speaker's seat in the virtual conference room can be determined according to the second voiceprint information. The embodiment of the present application can simulate the characteristics of the sound source of the speaker, so that the voice of the speaker can be recognized, and the user can clearly distinguish the speech content of each speaker.

100: Audio processing device

200: server

300: Electronic equipment

310: master device

320: slave device

210: first processor

220: The first memory

311: second processor

312: Second memory

313:The first audio module

314: The first display screen

315: The first front camera

321: the third processor

322: The third memory

323:Second audio module

324: Second display screen

325: Second front camera

S101-S104, S201-S207, S301-S307, S401-S403: steps

FIG. 1 is a schematic structural diagram of an audio processing device according to an embodiment of the present application.

FIG. 2 is a flowchart of an audio processing method in an embodiment of the present application.

Fig. 3a is a schematic structural diagram of a virtual conference room according to an embodiment of the present application.

Fig. 3b is a schematic structural diagram of a virtual meeting room according to another embodiment of the present application.

FIG. 4 is a flowchart of an audio processing method in another embodiment of the present application.

FIG. 5 is a schematic diagram of a virtual meeting room according to an embodiment of the present application.

FIG. 6 is a schematic diagram of a virtual conference room in another embodiment of the present application.

FIG. 7 is a flowchart of an audio processing method in another embodiment of the present application.

FIG. 8 is a schematic diagram of the first voiceprint information in an embodiment of the present application.

FIG. 9 is a flowchart of an audio processing method in another embodiment of the present application.

It should be noted that "at least one" in the embodiments of the present application refers to one or plural, and "plural" refers to two or more than two. "And/or" describes the relationship between related objects, and means that there may be three relationships. For example, A and/or B can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and scope of claims of this application and the drawings are used to distinguish similar items, rather than to describe a specific order or sequence.

In addition, it should be noted that the method disclosed in the embodiment of the application or the method shown in the flow chart includes one or a plurality of steps for realizing the method. Without departing from the scope of the claim, the execution order of the plurality of steps can be interchanged with each other, and some of the steps can also be deleted.

FIG. 1 is a schematic structural diagram of an audio processing device 100 according to an embodiment of the present application.

Referring to FIG. 1 , the audio processing device 100 may include a server 200 and an electronic device 300 . The electronic device 300 includes a master device 310 and a slave device 320 . The master device 310 refers to the electronic device used by the conference host, and the slave device 320 refers to the electronic device used by the other participants. The server 200 is communicatively connected to the master device 310 and the slave device 320 . The moderator initiates a virtual conference through the master device 310 , the server 200 builds a virtual conference room according to the instructions from the master device 310 , the master device 310 sends a meeting link to the slave device 320 , and other participants enter the virtual conference room through the slave device 320 .

Wherein, the communication connection may include a wired connection and a wireless connection. Wired connection refers to connection through wired transmission media such as optical fiber or twisted pair. Wireless connection refers to connection through wireless transmission media such as WiFi or mobile communication network (such as 2G/3G/4G/5G).

In some embodiments, the audio processing device 100 may also include a 360-degree fish-eye camera (not shown in the figure). A 360-degree fish-eye camera refers to a panoramic camera that can independently monitor a large area without blind spots. The 360-degree fisheye camera is communicatively connected to the server 200. The 360-degree fisheye camera can be set above some workstations in the office, with the lens pointing up or down, to photograph some workstations in the office. The server 200 maps the workstations captured by the 360-degree fisheye camera to the virtual conference room model, so that the people at the workstations feel as if they are in the virtual conference room. When the picture captured by the camera is an inverted picture, the server 200 performs inversion processing on the pictures displayed by the master device 310 and the slave device 320 to correct the direction of the picture.

The server 200 may include a first processor 210 and a first memory 220, and the first processor 210 may run computer programs or codes stored in the first memory 220 to implement the audio processing method of some embodiments of the present application.

The first processor 210 may include one or a plurality of processing units. For example, the first processor 210 can include, but it is not limited to the Application Processor (AP), the modem processor, the graphics processor, the Graphics Processing Unit (GPU), the image signal processor (IMP), and the control. Makers, video transcoditors, and digital signal processors (DSP), baseband processors, neuros network processors, NPUs, NPUs, etc. Wherein, different processing units may be independent devices, or may be integrated in one or multiple processors.

A memory may also be provided in the first processor 210 for storing instructions and data. In some embodiments, the memory in the first processor 210 is a cache memory. The memory can store instructions or data that the first processor 210 has just used or used repeatedly. If the first processor 210 needs to use the instruction or data again, it can be called directly from the memory.

In some embodiments, the first processor 210 may include one or more interfaces. The interface may include, but not limited to, an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver/Transmitter (UART) interface, and a Mobile Industry Processor Interface (Mobile Industry Processor Interface, MIPI), General-Purpose Input/Output (GPIO) interface, Subscriber Identity Module (SIM) interface, Universal Serial Bus (USB) interface, etc.

It can be understood that the interface connection relationship between the various modules shown in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation of the server 200 . In other embodiments of the present application, the server 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

The first memory 220 may include an external memory interface and an internal memory. Wherein, the external memory interface can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the server 200 . The external memory card communicates with the first processor 210 through the external memory interface to realize the data storage function. Internal memory may be used to store computer-executable code, which includes instructions. The internal memory may include a program storage area and a data storage area. Wherein, the stored program area can store the operating system, at least one application program required by a function (such as sound playing function, image playing function, etc.) and the like. The data storage area can store data (such as audio data, image data, etc.) created during the use of the server 200 . In addition, the internal memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory device, flash memory device, or Universal Flash Storage (UFS) and the like. The first processor 210 executes the instructions stored in the internal memory, and/or the instructions stored in the memory provided in the first processor 210, Execute various functional applications and data processing of the server 200, such as realizing the audio processing methods of some embodiments of the present application.

In some embodiments, the server 200 may include multiple virtual machines (Virtual Machine, VM). The server 200 has high availability (High Availability, HA) and auto scaling (Auto Scaling) functions. High availability refers to the ability to provide redundant processing. When a node (Node) is unavailable or unable to process a user's request, the request will be transferred to another available node in time for processing. The elastic scaling function refers to the ability to automatically adjust computing power (that is, the number of instances) according to business needs and policies. When business demands grow, auto-scaling automatically increases instances of the specified type to ensure computing power. When business demand drops, auto-scaling automatically reduces instances of the specified type to save costs.

In some embodiments, the main device 310 may include a second processor 311 , a second memory 312 , a first audio module 313 and a first display 314 . The second processor 311 is electrically connected to other above-mentioned components and the first processor 210 of the server 200 . The first audio module 313 is used for analog-to-digital conversion, encoding and decoding of audio signals. The first display screen 314 is used to display the scene of the virtual conference room and the avatars of some participants. The second processor 311 can run computer programs or codes stored in the second memory 312 to implement the audio processing methods of other embodiments of the present application.

In this embodiment, the main device 310 may include 3 degrees of freedom (Degree of Freedom, DoF) virtual reality (Virtual Reality, VR) glasses or a head-mounted device (Head-Mounted Device, HMD) that can track the user's eyeball movement direction.

It can be understood that, the specific implementation manners of the second processor 311 and the second memory 312 can refer to the above-mentioned first processor 210 and the first memory 220 , which will not be repeated here.

In some embodiments, the first audio module 313 can be installed in the second processor 311 , or some functional modules of the first audio module 313 can be installed in the second processor 311 . The main device 310 can implement audio functions, such as voice playback and recording, through the first audio module 313 .

In other embodiments, the main device 310 may further include a first front camera 315 . The first front camera 315 is electrically connected to the second processor 311 . The first front camera 315 is used to take pictures of people's faces and capture the movement direction of people's eyes, so as to support the server 200 to analyze the degree of concentration of the users participating in the virtual meeting and the degree of interest in the meeting topics.

In this embodiment, the main device 310 may include a smart phone, a tablet computer, a personal computer (Personal Computer, PC) or a personal digital assistant (Personal Digital Assistant, PDA).

In some embodiments, the slave device 320 may include a third processor 321 , a third memory 322 , a second audio module 323 , a second display screen 324 and a second front camera 325 . The third processor 321 is electrically connected to other above-mentioned components, the first processor 210 of the server 200 and the second processor 311 of the main device 310 .

It can be understood that the components and specific implementation of the slave device 320 can refer to the master device 310 .

The structure shown in the embodiment of this application does not constitute a specific limitation on the server 200 , the master device 310 or the slave device 320 . In other embodiments of the present application, the server 200 , the master device 310 or the slave device 320 may include more or less components than shown in the figure, or combine some components, or separate some components, or arrange different components. The illustrated components can be implemented in hardware, software or a combination of software and hardware.

FIG. 2 is a flowchart of an audio processing method in an embodiment of the present application.

Referring to FIG. 2, the audio processing method of this embodiment is applied to the master device 310, and the audio processing method may include the following steps:

S101 , in response to the host's first operation, the main device 310 sends a request to establish a conference to the server 200 .

Wherein, the first operation may include triggering a control item of setting up a meeting in the three-dimensional graphics software (such as Blender) of the main device 310 .

In this embodiment, 3D graphics software is installed on the main device 310 , and the moderator can trigger the control item of establishing a conference in the 3D graphics software, so that the main device 310 sends a request for establishing a conference to the server 200 . Among them, 3D graphics and image software can provide comprehensive 3D creation tools, including modeling (Modeling), UV-Mapping (UV-Mapping), texture (Texturing), binding (Rigging), skinning (Skinning), animation (Animation), particle (Particle) and physics analogy (Physics), scripting control (Scripting), rendering (Rendering), motion tracking (Motion Tracking), synthesis (Compositing), post-processing (Post-pro) reduction) and so on.

S102 , in response to the host's second operation, the main device 310 selects a virtual conference room model from the model library of the server 200 .

Wherein, the second operation may include triggering the selection of the control item of the virtual conference room model in the three-dimensional graphics software of the main device 310 .

Referring to FIG. 3a and FIG. 3b, the model library stores virtual conference rooms with different shapes and characteristics, such as cuboid virtual conference rooms, circular virtual conference rooms, and the like. The server 200 allows the main device 310 to access the model library of the server 200 after receiving the meeting establishment request from the main device 310 . The moderator can select the virtual conference room model in the three-dimensional graphics software, and trigger the control item for selecting the virtual conference room model, so that the main device 310 selects the virtual conference room model from the model library of the server 200 .

S103 , in response to the host's third operation, the main device 310 collects the first voiceprint information of the host, and sends the first voiceprint information to the server 200 .

Wherein, the third operation may include triggering a control item for recording audio in the three-dimensional graphic image software of the host device 310 .

In this embodiment, after the main device 310 determines the model of the virtual meeting room, the moderator triggers the control item of recording audio in the three-dimensional graphics software of the main device 310, the main device 310 records the voice signal of the moderator through the first audio module 313, and extracts the first voiceprint information of the moderator from the voice signal, And send the first voiceprint information to the server 200. Wherein, the first voiceprint information may include the frequency, amplitude and phase difference of the voice signal.

S104 , in response to the fourth operation of the moderator, the master device 310 determines the seats of the participants according to the virtual conference room model, and sends a conference link to the slave device 320 .

Wherein, the fourth operation may include triggering a control item of adding a conference link in the three-dimensional graphics software of the main device 310 .

In this embodiment, after the host device 310 sends the host's first voiceprint information to the server 200, the host triggers the control item of adding a conference link in the three-dimensional graphics software of the host device 310. The host device 310 determines the participants' seats according to the virtual conference room model, and each seat in the virtual conference room corresponds to a unique conference link. The master device 310 sends the conference link to the slave device 320 .

For example, N seats can be set in the virtual conference room, and one seat corresponds to one conference link, and the master device 310 can send the N conference links to N slave devices 320 respectively. After the participants enter the virtual meeting room through a meeting link, they can observe the virtual meeting room and other participants from the perspective of the corresponding seats, and can speak during the meeting.

In some embodiments, multiple observer seats can also be set in the virtual conference room, and each observer seat also corresponds to a unique conference link. The master device 310 can send the conference links of the M observation seats to the M slave devices 320 respectively. After the participants enter the virtual meeting room through a meeting link, they can observe the virtual meeting room and other participants from the perspective of the observer seats, and the participants in the observer seats cannot speak. Wherein, both M and N are positive integers.

In other embodiments, the master device 310 sends the conference link of the virtual meeting room to the plurality of slave devices 320 . Participants can select seats through the slave device 320 after entering the virtual meeting room through the meeting link of the virtual meeting room.

FIG. 4 is a flowchart of an audio processing method in another embodiment of the present application.

Referring to FIG. 4, the audio processing method of this embodiment is applied to the slave device 320, and the audio processing method may include the following steps:

S201 , the slave device 320 receives a conference link from the master device 310 .

In this embodiment, after the virtual meeting room is established, the main device 310 sends the meeting link of the virtual meeting room or the meeting link of the seats in the virtual meeting room to the participants. Participants receive a conference link from the master device 310 through the slave device 320 .

S202, in response to the first operation of the participant, the slave device 320 enters the virtual conference room according to the conference link.

Wherein, the first operation may include clicking a meeting link on the slave device 320 to start a browser application (such as Chrome Browser).

Referring to FIG. 5 , after the slave device 320 receives the conference link from the master device 310 , the participant clicks the conference link on the slave device 320 to start the browser application and enter the virtual conference room through the browser application.

S203, the slave device 320 determines whether the conference link is a conference link with reserved seats according to whether there are seats in the virtual conference room. If the conference link is a reserved seat conference link, step S204 is performed. If the conference link is not a conference link for seat reservation, step S205 is executed.

In this embodiment, after the slave device 320 enters the virtual conference room according to the conference link, the slave device 320 determines whether the conference link is a conference link with reserved seats according to whether there are seats in the virtual conference room. When a participant has a seat in the virtual meeting room, the slave device 320 displays the scene of the virtual meeting room and other participants from the perspective of the reserved seat. When the participants have no seats in the virtual conference room, the slave device 320 displays the overall virtual conference room scene. The slave device 320 can determine whether the participant has a seat in the virtual meeting room according to the different viewing angles of the participant after entering the virtual meeting room, and then determine whether the meeting link is a meeting link with reserved seats.

S204, in response to the second operation of the participant, collect the first voiceprint information of the participant from the device 320, and send the first voiceprint information to the server 200.

Wherein, the second operation may include triggering a control item for recording audio in the browser application of the slave device 320 .

In this embodiment, after the slave device 320 determines that the conference link is the conference link for reserved seats, the participant triggers the control item of recording audio in the browser application of the slave device 320, and the slave device 320 records the voice signal of the participant through the second audio module 323, extracts the first voiceprint information of the participant from the voice signal, and sends the first voiceprint information to the server 200.

S205 , in response to the third operation of the participant, determine the seat from the device 320 .

Wherein, the third operation may include triggering a control item of seat selection in the browser application of the slave device 320 .

In this embodiment, after the slave device 320 determines that the conference link is not a conference link for seat reservation, the participant triggers the seat selection control item in the browser application of the slave device 320, selects a seat from the slave device 320, and reads the selected information of the seat. Wherein, the seat selection information includes the seat has been selected or the seat has not been selected.

S206, the slave device 320 determines whether the seat has been selected by other participants. If the seat has been selected by other participants, return to step S205. If the seat is not selected by other participants, return to step S204.

In this embodiment, after a participant selects a seat through the slave device 320, the slave device 320 can read the selection information of the seat to determine whether the seat has been selected by other participants. When the seat selected by the participant is not selected by other participants, the slave device 320 can display the scene of the virtual conference room and other participants from the perspective of the selected seat. When the seat selected by the participant has been selected by other participants, the slave device 320 prompts the participant to reselect the seat.

S207, the slave device 320 displays a bitmap of seats in the virtual conference room.

In this embodiment, after the slave device 320 transmits the first voiceprint information to the server 200, the seat dot matrix of the virtual conference room is displayed. Please refer to Figure 6, there are 6 seats in the circular virtual meeting room, and the 6 seats are surrounded to form a dot matrix of the virtual meeting room, presenting the effect of a real virtual meeting room.

FIG. 7 is a flowchart of an audio processing method in another embodiment of the present application.

Referring to FIG. 7, the audio processing method of this embodiment is applied to the server 200, and the audio processing method may include the following steps:

S301, the server 200 receives a request from the master device 310 to establish a meeting.

In this embodiment, the moderator can trigger the control item of establishing a meeting in the 3D graphics software, so that the main device 310 sends a request for establishing a meeting to the server 200 . The server 200 receives a conference establishment request from the main device 310 .

S302, the server 200 releases the access authority of the model library to the main device 310 according to the request for establishing a meeting.

In this embodiment, after the server 200 receives the meeting establishment request from the main device 310, it opens the access authority of the model library to the main device 310, allowing the main device 310 to access the model library of the server 200 and call the virtual meeting room model from the model library.

S303, the server 200 creates a virtual meeting room according to the virtual meeting room model selected by the main device 310.

In this embodiment, the moderator can select a virtual conference room model in the 3D graphics software, and trigger the control item for selecting the virtual conference room model, so that the main device 310 selects the virtual conference room model from the model library of the server 200 . The server 200 establishes a virtual conference room according to the virtual conference room model selected by the host device 310 .

In some embodiments, the server 200 can create a virtual conference room model according to the preset ratio of the virtual conference room, and through the UV mapping tool, the master device 310 or the slave device 320 can display the dynamic image of the virtual conference room model.

In some other embodiments, the server 200 can extract texture features from the virtual meeting room picture according to the pre-stored virtual meeting room picture, and add the texture feature to the preset basic model through a mapping tool, so that the master device 310 or the slave device 320 can display the static picture of the virtual meeting room model.

S304, the server 200 sets the number of grid vertices according to the seat distribution of the virtual conference room.

Among them, Mesh is the basic unit of three-dimensional graphics and image software composition, and the virtual conference room is composed of multiple meshes. A grid includes 4 vertices (Vertex). The more the number of grid vertices contained in the area covered by a seat in the virtual conference room, the greater the density of the grid vertices in this area. The server 200 sets the number of grid vertices according to the seat distribution of the virtual meeting room, and sets different numbers of grid vertices in the area covered by each seat, that is, the density of the grid vertices in the area covered by each seat is different, so that the number or density of the seats and the grid vertices form a one-to-one correspondence.

S305, the server 200 receives the first voiceprint information from the master device 310 or the slave device 320 .

Wherein, the first voiceprint information may include the frequency, amplitude and phase difference of the voice signal.

In this embodiment, when the host speaks, the host triggers the audio recording control item in the three-dimensional graphics software of the main device 310. The main device 310 records the voice signal of the host through the first audio module 313, extracts the first voiceprint information of the host from the voice signal, and transmits the first voiceprint information to the server 200. The server 200 can receive the first voiceprint information from the main device 310 .

When a participant speaks, the participant triggers the audio recording control item in the browser application of the slave device 320, and the slave device 320 records the participant's voice signal through the second audio module 323, extracts the participant's first voiceprint information from the voice signal, and sends the first voiceprint information to the server 200. The server 200 can receive the first voiceprint information from the slave device 320 .

S306, the server 200 adjusts the frequency or amplitude of the first voiceprint information according to the number of grid vertices to obtain the second voiceprint information.

In this embodiment, each seat in the virtual conference room has a corresponding number of grid vertices. The server 200 adjusts the frequency or amplitude of the first voiceprint information according to the number of grid vertices. For example, seats with more grid vertices or higher density correspond to higher frequencies or larger amplitudes of the first voiceprint information.當第一座位所覆蓋區域之網格頂點之數目n ₁與第二座位所覆蓋區域之網格頂點之數目n ₂滿足：n ₁ >n ₂時，伺服器200調整來自於第一座位之第一聲紋資訊或來自於第二座位之第一聲紋資訊，使得來自於第一座位之第一聲紋資訊和來自於第二座位之第一聲紋資訊滿足：f ₁ >f ₂或a ₁ >a ₂ ，其中，f ₁表示來自於第一座位之第一聲紋資訊之頻率，f ₂表示來自於第二座位之第一聲紋資訊之頻率，a ₁表示來自於第一座位之第一聲紋資訊之振幅，a ₂表示來自於第二座位之第一聲紋資訊之振幅。

For example, referring to FIG. 8, the server 200 presets the number of grid vertices for each seat. When the server 200 receives the first voiceprint information of 6 participants, it can process the 6 pieces of first voiceprint information, adjust the frequency or amplitude of the corresponding first voiceprint information according to the number or density of the grid vertices of each seat, and obtain 6 pieces of second voiceprint information, so as to improve the recognizability of the voiceprint information.

S307, the server 200 determines the speaker's seat in the virtual conference room according to the second voiceprint information.

In this embodiment, after the server 200 captures the first voiceprint information, the source of the first voiceprint information cannot be determined. The server 200 establishes a correspondence between the number of grid vertices in the area covered by each seat in the virtual conference room and the first voiceprint information, and the area with more grid vertices corresponds to a higher frequency or amplitude of the first voiceprint information. The server 200 adjusts the frequency or amplitude of the first voiceprint information from different seats according to the number of grid vertices to obtain more recognizable second voiceprint information. Since the frequency or amplitude of the second voiceprint information on each seat is different, the second voiceprint information is different from each seat. With a one-to-one correspondence, the server 200 can determine the speaker's seat in the virtual conference room according to the second voiceprint information.

FIG. 9 is a flowchart of an audio processing method in another embodiment of the present application.

Referring to FIG. 9, the audio processing method of this embodiment is applied to the server 200, and the audio processing method may include the following steps:

S401, the server 200 controls the slave device 320 to collect eye movement direction information of the participants.

In this embodiment, when a participant is speaking, the server 200 controls other slave devices 320 to collect eye movement direction information of other participants after identifying the voiceprint information of the participant.

S402, the server 200 determines the participant's concentration according to the participant's eyeball movement direction information.

Among them, the degree of concentration refers to the degree of concentration of the participants on the content of the speaker's speech or the degree of concentration on the topic of the meeting. The higher the degree of concentration, the more interested the participants are in the meeting topics. When a speaker is speaking, the server 200 can determine the concentration of other participants according to the eye movement direction information of other participants after receiving the eye movement direction information of other participants.

For example, when a speaker is speaking, if a participant's eyeball moves towards the speaker, it means that the participant is currently concentrating, and the concentration of the participant can be marked as 1. If the participant's eye movement direction is far away from the speaker, it means that the participant is currently not paying attention, and the concentration of the participant can be marked as 0. In the 10 rounds of speeches in the whole meeting, if the number of times a participant's concentration is 1 is 6 rounds, and the number of times of concentration is 0 is 4 rounds, it can be considered that the participant's concentration on the conference topic is 6/10=0.6.

S403, the server 200 determines whether the participant is interested in the conference topic according to the concentration of the participant.

In this embodiment, the server 200 counts the concentration of the participants on the conference topic, and can compare the concentration with a preset interest threshold. If the concentration is greater than or equal to the interest threshold, it means that the participant is interested in the conference topic. If the concentration is less than the interest threshold, it means that the participants are not interested in the meeting topic.

For example, the preset interest threshold is 0.6. In the whole meeting, if a participant's concentration on the conference topic is 0.5, since the participant's concentration on the conference topic is less than the interest threshold, it means that the participant is not interested in the conference topic. If a participant's concentration on the conference topic is 0.7, since the participant's concentration on the conference topic is greater than the interest threshold, it means that the participant is interested in the conference topic.

The embodiment of the present application also provides a storage medium for storing computer programs or codes, and when the computer programs or codes are executed by a processor, the audio processing method of the embodiments of the present application is implemented.

Storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Storage media include, but are not limited to, random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), flash memory or other memories, compact disc read-only memory (CD-ROM), digital versatile disc ( Digital Versatile Disc (DVD) or other optical disc storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store desired information and can be accessed by a computer.

The embodiments of the present application have been described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge of ordinary skilled persons in the technical field without departing from the purpose of the present application.

S301-S307: Steps

Claims (9)

An audio processing method for a virtual meeting room, comprising: setting the number of grid vertices according to the seat distribution of the virtual meeting room, specifically including: setting different numbers of the grid vertices in the area covered by each seat, so as to establish a corresponding relationship between the seat and the number of grid vertices; extracting the first voiceprint information of the speaker, the first voiceprint information including the frequency, amplitude and phase difference of the voice signal; adjusting the frequency or amplitude of the first voiceprint information according to the number of grid vertices to obtain second voiceprint information; Seats in a virtual conference room. The audio processing method according to claim 1, wherein said adjusting the frequency or amplitude of the first voiceprint information according to the number of grid vertices includes: when the number of grid vertices in the area covered by the first seat is greater than the number of grid vertices in the area covered by the second seat, increasing the frequency of the first voiceprint information from the first seat, or lowering the frequency of the first voiceprint information from the second seat, so that the frequency of the first voiceprint information from the first seat is greater than the frequency of the first voiceprint information from the second seat. The audio processing method according to claim 1, wherein the adjusting the frequency or amplitude of the first voiceprint information according to the number of grid vertices includes: when the number of grid vertices in the area covered by the first seat is greater than the number of grid vertices in the area covered by the second seat, increasing the amplitude of the first voiceprint information from the first seat, or reducing the amplitude of the first voiceprint information from the second seat, so that the amplitude of the first voiceprint information from the first seat is greater than the amplitude of the first voiceprint information from the second seat. The audio processing method according to claim 1, wherein, after determining the seat of the speaker in the virtual conference room according to the second voiceprint information, the party The method also includes: extracting the eye movement direction information of the participant; determining the concentration of the participant according to the eye movement direction information, and the concentration value is 0 or 1; determining whether the participant is interested in the conference topic according to the concentration. The audio processing method according to claim 4, wherein said determining the concentration of the participant according to the eye movement direction information includes: marking the concentration as 1 when the eye movement direction of the participant is toward the speaker; marking the concentration as 0 when the eye movement direction of the participant is away from the speaker. The audio processing method as described in claim 5, wherein the method further includes: when there are multiple speakers, counting the value of the concentration of the participants when each speaker speaks; and determining the concentration of the participants on the conference topic according to the value of the concentration. The audio processing method as described in claim 4, wherein the determining whether the participant is interested in the conference topic according to the degree of concentration includes: determining that the participant is interested in the conference topic when the value of the concentration degree is greater than or equal to a preset interest threshold; determining that the participant is not interested in the conference topic when the value of the concentration degree is smaller than the interest threshold. An audio processing device, including a server, a master device and a slave device, the master device is used to initiate a virtual meeting, and the server is used to A virtual conference room is constructed, the slave device is used to enter the virtual conference room according to the link from the master device, the server includes a first processor and a first memory, and the improvement is that the first processor runs a computer program or code stored in the first memory to implement the audio processing method as described in any one of claims 1 to 7. A storage medium for storing computer programs or codes, the improvement is that when the computer programs or codes are executed by a processor, the audio processing method as described in any one of Claims 1 to 7 is realized.

Images