TW201901527A - Video conference and video conference management method - Google Patents

Abstract

A video conference device and corresponding video conference management method are provided. The method includes receiving a sound occurred in a conference space; determining a first position of the sound according to the received sound; capturing a panoramic image of the conference space; identifying face images of a plurality of participants in the panoramic image, and identifying second positions of the face images in the panoramic image; determining a speaker among the participants according to the first position, the second positions and the face images; setting the panoramic image to display in a first region of a video conference image, enlarging the determined speaker's image of the panoramic image, and setting the enlarged speaker's image to display in a second region of the video conference image.

Description

Video conference device and video conference management method

The invention relates to a video conference device, and more particularly to a video conference device and a video conference management method suitable for a video conference.

The traditional video conference system uses more than three cameras to shoot people participating in the conference, and simultaneously uses a microphone array to locate the speakers, and enlarges the positioned speakers in the video conference image. However, the traditional method only performs sound localization to determine the position of the sound source, and considers the position of the sound source to be the position of the speaker, and then enlarges the image of the position in the video conference image. Therefore, the above-mentioned traditional method may have insufficient accuracy due to the image of environmental noise, and cannot accurately determine the position of the speaker.

The invention provides a video conference device and a video conference management method, which can accurately and automatically detect a speaker through sound localization and image recognition, so as to enlarge and display the speaker's image in the video conference image.

An embodiment of the present invention provides a video conference device. The device includes a microphone array, a sound positioning unit, an image capturing device, an image recognition unit, and a video conference management unit. The microphone array includes a plurality of microphones, and is configured to receive sounds emitted from a conference space around the video conference device. The sound positioning unit is coupled to the microphone array, and is configured to determine a first position of the sound according to the received sound. The image capturing device is used for capturing a panoramic image of the conference space. The image recognition unit is coupled to the image capture device to identify a facial image of at least one participant in the panoramic image, and identify a second of the at least one facial image in the panoramic image. position. The video conference management unit is coupled to the sound positioning unit, the image recognition unit, and the storage unit, and is configured to be based on the first position, the at least one second position, and the at least one face image. To determine a speaker of the at least one participant. In addition, the video conference management unit sets the panoramic image to be displayed in a first area of the video conference image, enlarges the image of the speaker determined in the panoramic image, and sets the enlarged speaker's image. The image is displayed in a second area of the video conference image.

In the above embodiment, the video conference management unit performs a voice-to-text operation on the sound issued by the speaker to convert the sound of the speaker into a text message corresponding to the speaker , Wherein the video conference management unit stores the text message corresponding to the speaker to a conference record database.

An embodiment of the present invention provides a video conference management method suitable for a video conference in a conference space, wherein the conference space has at least one participant. The method includes receiving a sound emitted from the conference space; determining a first position of the sound based on the received sound; capturing a panoramic image of the conference space; identifying a panoramic image in the panoramic image A face image of the at least one participant, and identifying a second position of the at least one face image in the panoramic image; according to the first position, the at least one second position, and the at least one A face image to determine a speaker in the at least one participant; and setting the panoramic image to be displayed in a first area of a video conference image, enlarging the image of the determined speaker in the panoramic image, and The enlarged image of the speaker is set to be displayed in a second area of the video conference image.

In the above embodiment, the video conference management unit performs a voice-to-text operation on the sound issued by the speaker to convert the sound of the speaker into a text message corresponding to the speaker , Wherein the video conference management unit stores the text message corresponding to the speaker to a conference record database.

Based on the above, the video conference device and the video conference management method provided by the present invention can accurately identify the speakers in the conference space where the video conference is held by using sound localization and image recognition, and enlarge and display the images of the speakers on the In a video conference image having panoramic images of all participants in the conference space. In addition, a speech-to-text operation can be performed on a speaker's speech, and a text message corresponding to the speaker's identification name and the speech is stored to establish a meeting record of the video conference. In this way, the video conference device and video conference management method provided by the present invention can allow all participants to intuitively and focus on the speaker to conduct the video conference more efficiently, and the device and method can also instantly Establish meeting records, which in turn improves the overall efficiency of video conferences.

In order to make the above features and advantages of the present invention more comprehensible, embodiments are hereinafter described in detail with reference to the accompanying drawings.

FIG. 1A is a schematic diagram of a video conference according to an embodiment of the invention. FIG. 1B is a schematic diagram illustrating a panoramic image corresponding to the video conference in FIG. 1A according to an embodiment of the present invention. Please refer to FIGS. 1A and 1B at the same time. Assume that in conference space 1 (such as a conference room or lecture hall), four participants 2, 3, 4, and 5 are holding a video conference (or other video conference Conference, video teaching, etc.), and the video conference device 10 is arranged in the conference space 1. The video conference device 10 provided in this embodiment captures surrounding images to obtain a panoramic image 11 (for example, the video conference device 10 obtains a panoramic image through 360-degree panoramic photography). As shown in FIG. 1B, the panoramic image 11 will include all participants 2, 3, 4, and 5, and the panoramic image 11 will also include conference spaces around the participants 2, 3, 4, and 5. Image. In this embodiment, the video conference device 10 judges who is currently speaking among all the participants, and generates (outputs) a video conference image based on the judgment result. For example, when the participant 2 speaks (ie, the participant 2 is a speaker), the video conference device 10 receives the sound 21 (also referred to as a speech 21) from the participant 2, and according to the received sound 21 Perform voice localization, and determine that the participant 2 is the speaker who made the above-mentioned voice (speech) according to the change in the facial image of the participant 2 and the obtained voice localization result. Hereinafter, the video conference device provided by the present invention will be described in detail with reference to FIG. 2.

FIG. 2 is a block diagram of a video conference device according to an embodiment of the present invention. Please refer to FIG. 2. In this embodiment, the video conference device 10 includes a video conference management unit 110, a microphone array 120, a sound positioning unit 121, an image capture device 130, an image recognition unit 131, a storage unit 140, and a connection interface unit 150. . The sound positioning unit 121 is coupled to the microphone array 120. The image recognition unit 131 is coupled to the image capture device 130. The video conference management unit 110 is coupled to a sound positioning unit 121, an image recognition unit 131, a storage unit 140, and a connection interface unit 150.

In this embodiment, the video conference management unit 110 is hardware (for example, a chipset, a processor, and the like) with computing capabilities, and is used to control functions of other components of the video conference device 10 and manage the entirety of the video conference device 10 Operation. In this embodiment, the video conference management unit 110 is, for example, a central processing unit (CPU), a central processing unit (CPU), a graphic processing unit (GPU), or a micro-processor. Or other programmable processing unit (Microprocessor), digital signal processor (DSP), programmable controller, Application Specific Integrated Circuits (ASIC), programmable logic Programmable Logic Device (PLD) or other similar devices.

The storage unit 140 may temporarily store data according to an instruction of the video conference management unit 110. The data includes data for managing the video conference device 10, data received from other electronic devices, and data for transmitting to other electronic devices. Data or other types of data, the invention is not limited to this. In addition, in this embodiment, the storage unit 140 may also record some data that needs to be stored for a long time, such as storing multiple databases, through instructions of the video conference management unit 110. The multiple databases include a face database 141 and a meeting record database 142. In another embodiment, the plurality of databases further includes a voice database 143. It should be noted that the multiple databases may also be stored in a remote server and accessed via a communication connection (or network connection) with the video conference device. The face database 141 records a plurality of facial images corresponding to different people. The face database 141 may also record facial image feature data sets corresponding to multiple facial images of different people. The facial image feature data set records a plurality of image feature values of a corresponding facial image. In addition, the face database 141 may also record identification names (eg, names, codes, or identification codes) of persons corresponding to the plurality of facial images. The conference record database 142 records the content of each meeting. In particular, the content may include text messages corresponding to the speeches of all speakers in each meeting. The voice database 143 can record multiple voice messages of different people. In addition, the voice database 143 may also record a plurality of voice feature data groups respectively corresponding to the voices of the different persons. The voice feature data set records a plurality of voice feature values of the corresponding human voice.

The microphone array 120 includes a plurality of microphones configured on the video conference apparatus 10. The spatial range of the plurality of microphones receiving sound covers the space around the video conference device 10. The microphone array 120 converts the received sound into a sound signal for each microphone and transmits the sound signal to the sound positioning unit 121. Because the relative positions of each participant 2, 3, 4, 5 and the multiple microphones of the microphone array 120 are different. Therefore, it is assumed that the microphones of the microphone array 120 currently receive the voice spoken by the participant 2, and the sounds received by the microphones of the microphone array 120 will be different, so that the strength of the converted sound signals will also be different.

The sound positioning unit 121 is a circuit unit / chip that can calculate a position of the received sound according to a plurality of sound signals received from a plurality of microphones of the microphone array 120, respectively. In the above example, the sound localization unit 121 may calculate the sound source of the sound source according to the sounds of different sound intensity received from the same sound source (for example, the participant 2 of the above example) received by multiple microphones of the microphone array 120. Location (also known as the first location). The coordinates of the first position may be represented by a coordinate system (eg, a rectangular coordinate system or an angular coordinate system) corresponding to the panoramic image 11.

The image capturing device 130 is, for example, one or two cameras / cameras / lenses that can capture (capture) panoramic images. The image capturing device 130 can adjust the height. The panoramic image may include an image of a conference space around the video conference apparatus 10. The image recognition unit 131 sends the captured panoramic image to the image recognition unit 131.

The image recognition unit 131 is a circuit unit capable of performing an image recognition operation (such as a face detection operation and a face recognition operation) on the received image. The image recognition unit 131 may also be coupled to the storage unit 140 and perform machine learning by using multiple face images in the face database 141 to enhance the face detection operation performed by the image recognition unit 131 or Speed and accuracy of face recognition operations. The image recognition unit 131 may record the facial images (or corresponding facial image feature data sets) of the identified participants to the face database 141.

The connection interface unit 150 may be, for example, a Serial Advanced Technology Attachment (SATA) standard, a Parallel Advanced Technology Attachment (PATA) standard, or the Institute of Electrical and Electronic Engineers (IEEE). ) 1394 standard, Peripheral Component Interconnect Express (PCI Express) standard, Universal Serial Bus (USB) standard, Ultra High Speed-I (UHS-I) interface standard, Ultra High Speed-II (UHS-II) interface standard, Secure Digital (SD) interface standard, Memory Stick (MS) interface standard, Multi Media Card (MMC) Interface standards, Compact Flash (CF) interface standards, Integrated Device Electronics (IDE) standards, Personal Computer Memory Card International Association (PCMCIA) standards, video graphics Array (Video Graphics Array, VGA) standard, digital video Interface (Digital Visual Interface, DVI) standard, a high-definition multimedia interface sharpness (High Definition Multimedia Interface, HDMI) standard or other suitable standard circuit unit. In this embodiment, the video conference management unit 110 may be connected to a sound playback device 151 (eg, a speaker), a display device 152 (eg, a screen, a projector) or other types of output devices through the connection interface unit 150 to output Data (eg, video conference images generated by video conference device 10). In addition, the video conference management unit 110 may be connected to the input device 153 through the connection interface unit 150 to receive an input signal from the input device 153 or receive a control from a user (eg, a participant).

It should be noted that the output device and the input device can also be integrated into the same electronic device (eg, a touch screen). In particular, the connection interface unit 150 can also be connected to other storage units (such as a memory card, an external hard disk, etc.), so that the video conference management unit 110 can access the storage unit externally connected through the connection interface unit 150. data of. In addition, in another embodiment, different input / output devices connected through the connection interface unit 150 described above can also be integrated into the video conference device 10.

In an embodiment, the video conference device 10 may also be connected to other electronic devices (such as a desktop computer, a notebook computer, a tablet computer, a server, a smart phone, etc.) via the connection interface unit 150 to allow other The electronic device performs the video conference through the video conference device 10 and applications (such as communication software such as Skype, QQ, Line, FB Messenger, Google Handout, etc.) running in other electronic devices.

In another embodiment, the video conference device 10 further includes a communication unit 160 coupled to the video conference management unit 110. The communication unit 160 is configured to transmit or receive data through wireless communication. In this embodiment, the communication unit 160 may have a wireless communication module, and support a Global System for Mobile Communication (GSM) system, a Personal Handy-phone System (PHS), and multiple codes. Code Division Multiple Access (CDMA) system, Wireless Fidelity (WiFi) system, Worldwide Interoperability for Microwave Access (WiMAX) system, 3rd generation wireless communication technology (3G), One or a combination of fourth-generation wireless communication technology (4G), Long Term Evolution (LTE), infrared (Infrared) transmission, and Bluetooth (BT) communication technology, and is not limited thereto. In addition, the communication unit 160 may also have a Network Interface Card (NIC) to establish a network connection, so that the video conference device 10 can be connected to a local area network or the Internet.

In yet another embodiment, the video conference apparatus 10 further includes a voice recognition unit 122 coupled to the video conference management unit 110, the microphone array 120, and the storage unit 140. The voice recognition unit 122 is a circuit unit that can perform voice recognition operations on the sound received by the microphone array 120, and can be used to distinguish whether the sound is a human voice (voice). In addition, in the voice recognition operation performed, the voice recognition unit 122 can also identify the person who issued the voice against multiple voice messages or multiple voice feature data sets in the voice database 143 according to the recognized voice. The distinguished name of the. In addition, the voice recognition unit 122 can also perform a voice-to-text operation to convert the recognized voice (voice message) into a text message. It should be noted that the voice recognition unit 122 may perform machine learning by using multiple voice messages or multiple voice feature data sets in the voice database 143 to improve the performance of the voice recognition operations performed by the voice recognition unit 122. ability. The operation of the video conference device provided by this embodiment and the video conference management method used in this embodiment will be described in detail below with reference to FIGS. 3 and 4.

FIG. 3 is a flowchart of a video conference method according to an embodiment of the present invention. Please refer to FIG. 3, it is assumed that the participant 2 of the participants 2, 3, 4, and 5 (as shown in FIG. 1) currently makes a sound 21. In step S301, the microphone array 120 receives sounds emitted from the conference space. For example, a plurality of microphones of the microphone array 120 receive the sound 21. Next, in step S303, the sound positioning unit 121 determines a first position of the sound according to the received sound. That is, the sound positioning unit 121 performs a sound positioning operation according to a plurality of sound signals generated by the microphone array 120 receiving the sound 21 to calculate the position of the sound source that emits the sound 21.

It should be noted that, in an embodiment, if the voice recognition unit 122 determines that the received sound is not a human voice, step S303 will not be continued to process the received sound. In this way, the interference of non-human ambient noise can be avoided. In addition, as described above, if the voice recognition unit 122 determines that the received voice is a human voice, the voice recognition unit 122 or the video conference management unit 110 may perform a voice-to-text operation corresponding to the voice characteristics of the human voice received. And using a speech recognition model trained through a speech database to assist in correcting the converted text message. Then, use the recognized name of the speaker (for example, you can use the image recognition or speech recognition method to find the speaker's recognition name through the face database or voice database) and the text message record as the The speeches of the speakers of this meeting record are stored in the meeting record database.

In step S305, the image capturing device 130 captures a panoramic image of the conference space. As in the example of FIGS. 1A and 1B, the image capturing device 130 captures and generates a panoramic image 11 of the conference space 1, and transmits the image data corresponding to the panoramic image 11 to the image recognition unit 131.

In step S307, a facial image of at least one participant in the panoramic image is identified, and a second position of the at least one facial image is identified.

Specifically, the image recognition unit 131 sets a coordinate system for the received panoramic image 11. In addition, the image recognition unit 131 continuously detects whether there is a face image in the panorama image (via a face detection operation) on the received panorama image 11. If the face image of at least one participant (one or more participants) is detected, the image recognition unit 131 will detect the detected face image according to the position of the detected face image in the panoramic image. The facial image sets a coordinate value, and the coordinate value indicates the position of the detected facial image in the panoramic image. For example, this coordinate value can be used to represent the center point of the facial image, the center point of the mouth region of the facial image (eg, the center point 503 of the mouth region in FIG. 5A), or the corresponding point to cover the face The coordinates of a point in a specific area of the image. The present invention is not limited to the manner of setting coordinate values corresponding to facial images.

In addition, in an embodiment, the image recognition unit 131 attempts to perform face recognition on the detected facial image (via a face recognition operation). In the face recognition operation, the image recognition unit 131 compares the face database 141 with the detected face image. If there is a matching face image, it can find the matched face accordingly. The distinguished name of the participant to which the image belongs. In an embodiment, if there is no matching face image, the image recognition unit 131 may add the detected face image to the face database 141 (the corresponding identification name may be used to receive input from the user (Obtained by operation, obtained by voice recognition, or obtained by accessing conference information containing the identification names of all participants).

It is worth mentioning that the video conference management unit 110 performs the operations of steps S301 to S303 and the operations of steps S305 to S307 in parallel (synchronously). In other words, the video conference management unit 110 can simultaneously and continuously recognize the position of the sound source of the currently received sound, continuously capture the panoramic image and recognize the facial image in the panoramic image and the image corresponding to the detected facial image. position.

Next, in step S309, the video conference management unit 110 determines a speaker among the at least one participant according to the first position, the at least one second position, and the at least one facial image.

FIG. 4 is a flowchart of step S309 of the video conference method according to an embodiment of the present invention. FIG. 5A is a schematic diagram of a panoramic image according to an embodiment of the present invention. FIG. 5B is a schematic diagram of a feature recognition area according to an embodiment of the invention. Referring to FIG. 5A, there are four participants in the panoramic image 500 of FIG. 5A. As described above, the image recognition unit 131 recognizes the facial images of each participant and the corresponding second positions.

Please refer to FIG. 4 and FIG. 5A at the same time. In step S3091, the video conference management unit 110 sets a target area in the panoramic image according to the first position, and identifies a target area based on the target area and the at least one second position. At least one target facial image in the target area. For example, it is assumed that the current video conference management unit 110 determines the first position 502 corresponding to the sound according to the received sound. The video conference management unit 110 sets a target area 501 with the first position 502 as the center, and determines the position 503 of the target facial image in the target area based on the set target area and the identified second positions. And the corresponding target facial image. In more detail, the video conference management unit 110 determines whether at least one second position is covered in the target area according to the coverage of the target area (coordinate values corresponding to the area boundaries). If so, it is estimated that there will be a face image of the speaker in the target area.

Next, in step S3093, the video conference management unit 110 determines the speaker among the at least one target participant to which the at least one target facial image belongs according to the image change of the at least one target facial image.

For example, referring to FIG. 5B, the video conference management unit 110 or the image recognition unit 131 sets the reference coordinate values of the four corners of the target facial image 511. The length of the target facial image 511 is "H" and the width is "W". In this embodiment, the mouth area of the target facial image 511 may be set in advance as an area within the target facial image 511. Assume that the upper left corner of the target facial image 511 is O (0, 0); the upper right corner is W (W, 0); the lower left corner is H (0, H); the lower right corner is WH (W, H) . In this example, the range of the mouth area can be preset to the range of 3 / 5H to 4 / 5H and 1 / 3W to 2 / 3W of the target facial image. That is, the upper left corner of the mouth area (relative to the target facial image) is O1 (1 / 3W, 3 / 5H); the upper right corner is W1 (2 / 3W, 3 / 5H); the lower left corner is H1 (1 / 3W, 4 / 5H); the lower right corner is WH1 (2 / 3W, 4 / 5H).

In this embodiment, the video conference management unit 110 instructs the image recognition unit 131 to use the mouth area as a feature recognition area, and further calculates the image change for the feature recognition area. In more detail, the video conference management unit 110 instructs the image recognition unit 131 to set the feature recognition area 520 of the target facial image according to the target position 502 corresponding to the target facial image 511 of the plurality of second positions. After the feature recognition area 520 is set, the image recognition unit 131 calculates pixel change values in the feature recognition area 520 of the target facial image 511 over a period of time.

For example, at the time point of each video frame, the image recognition unit 131 calculates an average pixel value (eg, RGB value, grayscale value, brightness value, etc.) of the feature recognition area 520 of the panoramic image of the current video frame. Other types of pixel values). Next, the image recognition unit 131 calculates a difference (taken as an absolute value) between the average pixel value of the current video frame and the average pixel value of the feature recognition area 520 of the panoramic image of each of the first M video frames. Next, the image recognition unit 131 takes the largest of the plurality of differences as the pixel change value of the feature recognition area 520 corresponding to the current video frame.

Next, the image recognition unit 131 uses the calculated pixel change value corresponding to the target face image 511 as the characteristic image change value of the target face image 511. If the characteristic image change value exceeds a predetermined threshold, the video conference management unit 110 determines that the corresponding participant is a speaker.

It should be noted that in the above example, the target area has only one target participant. However, if the target area has multiple target participants, the image recognition unit 131 sets a feature recognition area corresponding to the facial images of all participants to calculate the characteristic image change values of all participants, and finds the largest one (also Called the maximum characteristic image change value). If the maximum characteristic image change value exceeds the predetermined threshold, the video conference management unit 110 determines that the participant corresponding to the maximum characteristic image change value is a speaker among the plurality of target participants.

Please return to FIG. 3 again. After the speaker is determined, in step S311, the video conference management unit 110 sets the panoramic image to be displayed in the first area of the video conference image, and enlarges the determined location in the panoramic image. The speaker's image is described, and the enlarged image of the speaker is set to be displayed in a second area of the video conference image.

5C is a schematic diagram of a video conference image according to an embodiment of the present invention. Referring to FIG. 5C, following the example in FIGS. 5A and 5B above, after determining that the leftmost participant in the panoramic image 500 is the speaker, the video conference management unit 110 will set the speaker according to the second position of the target facial image 511 Image (eg, image 510), and the video conference management unit 110 generates a video conference picture 530 according to the received panoramic image 500. For example, the generated video conference picture 530 will have a first area and a second area. The video conference management unit 110 sets the panoramic image 500 to be displayed in the first area. The video conference management unit 110 enlarges the speaker's image 510, and sets the enlarged image 510 to be displayed in the second area.

In one embodiment, the video conference management unit 110 may use machine learning to perform super-resolution operations, so that the enlarged image 510 of the speaker is kept clear (the operation of magnifying the image does not become blurred).

It should be noted that, in the above-mentioned example, the first region is disposed above the second region, but the present invention is not limited thereto. For example, in another embodiment, the first area may be disposed below the second area.

Please return to FIG. 3 again. After setting / generating the video conference image, in step S313, the video conference management unit 110 outputs the video conference image. Specifically, in this embodiment, the conference management unit 110 may convert the generated video conference image into a corresponding video signal through the connection established by the communication unit 160 and send it to other electronic devices, so that other electronic devices The screen or display device can display the resulting video conference image.

In an embodiment, the video conference device 10 may be connected to the electronic device in the conference space through the connection interface unit 150 or the communication unit 160 as a camera of the connected electronic device, so that the electronic device generally performs a video conference. Applications (such as instant messaging software such as Skype, Line, etc.) can conduct video conferences through video conference images generated by the video conference device 10. In this way, users can use common instant messaging software on the market to conduct smart video conferences. That is, the overall conference image (using the panoramic image of the first region of the video conference image) and the current speaker image (using the image of the second region of the video conference image) of the local end are provided to the user of the remote instant messaging software.

It is worth mentioning that, in another embodiment, the video conference management unit 110 may also attach the text message obtained through the voice-to-text operation to the generated video conference image in another layer so that the Text messages can be used as subtitles for speakers. In still another embodiment, the video conference management unit 110 may further input the text message obtained through a voice-to-text operation into a translation unit to obtain a translated text message (for example, a Chinese voice spoken by a speaker). Converted and translated into English text), and the translated text message is attached to the video conference video as corresponding subtitles.

In summary, the video conference device and the video conference management method provided by the present invention can accurately determine the speaker in the conference space where the video conference is held by using sound positioning and image recognition, and enlarge the image of the speaker and Displayed in a video conference image having panoramic images of all participants in the conference space. In addition, a speech-to-text operation can be performed on a speaker's speech, and a text message corresponding to the speaker's identification name and the speech is stored to establish a meeting record of the video conference. In this way, the video conference device and video conference management method provided by the present invention can allow all participants to intuitively and focus on the speaker to conduct the video conference more efficiently, and the device and method can also instantly Establish meeting records, which in turn improves the overall efficiency of video conferences.

Although the present invention has been disclosed as above with the examples, it is not intended to limit the present invention. Any person with ordinary knowledge in the technical field can make some modifications and retouching without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be determined by the scope of the attached patent application.

1‧‧‧ meeting space

2, 3, 4, 5‧‧‧ attendees

10‧‧‧ Video Conference Device

11, 500‧‧‧ panoramic image

21‧‧‧Voice / Voice / Speech

110‧‧‧Video Conference Management Unit

120‧‧‧Microphone Array

121‧‧‧ sound localization unit

122‧‧‧Speech recognition unit

130‧‧‧Image capture device

131‧‧‧Image recognition unit

140‧‧‧Storage unit

141, 142, 143‧‧‧ database

150‧‧‧ connect interface unit

151‧‧‧ sound playback device

152‧‧‧Display device

153‧‧‧input device

160‧‧‧communication unit

Process steps of S301, S303, S305, S307, S309, S311, S313‧‧‧ video conference management method

S3091, S3093‧‧‧‧ Steps in the video conference management method of FIG. 3

501‧‧‧target area

510‧‧‧Speaker image

511‧‧‧Target facial image

520‧‧‧Feature recognition area

H‧‧‧ Length of target facial image

W‧‧‧ The width of the target facial image

O (0, 0), W (W, 0), H (0, H), WH (W, H)

530‧‧‧Video conference video

FIG. 1A is a schematic diagram of a video conference according to an embodiment of the invention. FIG. 1B is a schematic diagram illustrating a panoramic image corresponding to the video conference in FIG. 1A according to an embodiment of the present invention. FIG. 2 is a block diagram of a video conference device according to an embodiment of the present invention. FIG. 3 is a flowchart of a video conference method according to an embodiment of the present invention. FIG. 4 is a flowchart of step S309 of the video conference method according to an embodiment of the present invention. FIG. 5A is a schematic diagram of a panoramic image according to an embodiment of the present invention. FIG. 5B is a schematic diagram of a feature recognition area according to an embodiment of the invention. 5C is a schematic diagram of a video conference image according to an embodiment of the present invention.

Claims (10)

A video conference device includes: a microphone array including a plurality of microphones for receiving a sound emitted from a conference space; a sound positioning unit coupled to the microphone array for receiving the sound according to the received sound To determine a first position of the sound; an image capture device to capture a panoramic image of the conference space; an image recognition unit coupled to the image capture device to identify at least one of the panoramic images A participant's face image, and determining a second position of the at least one face image; and a video conference management unit, coupled to the sound positioning unit and the image recognition unit, for using the first position, the At least a second position and the at least one face image to determine a speaker of the at least one participant, wherein the video conference management unit sets the panoramic image to be displayed in a first area of a video conference image and enlarges the panoramic image The image of the speaker determined in the image, and the enlarged image of the speaker is set to be displayed in a first portion of the video conference image Area. The video conference device according to item 1 of the scope of patent application, wherein the video conference management unit performs a voice-to-text operation on the voice issued by the speaker to convert the voice of the speaker into a corresponding speaker A text message, wherein the video conference management unit stores an identification name corresponding to the speaker and the text message to a conference record database. The video conference device according to item 1 of the scope of patent application, wherein the video conference management unit sets a target area in the panoramic image according to the first position, and instructs the image recognition unit according to the target area and the at least one The second position recognizes at least one target facial image in the target area, wherein the video conference management unit determines, among the at least one target participant to which the at least one target facial image belongs, according to an image change of the at least one target facial image. Of that speaker. The video conference device according to item 3 of the scope of patent application, wherein the video conference management unit instructs the image recognition unit to calculate a characteristic image change value of the at least one target facial image, and if one of the at least one characteristic image change value is a maximum The characteristic image change value exceeds a predetermined threshold, and the video conference management unit determines that the participant corresponding to the maximum characteristic image change value is the speaker of the at least one target participant. The video conference device according to item 4 of the scope of patent application, wherein the video conference management unit instructs the image recognition unit to set the at least one target position corresponding to at least one target position image of the at least one target face in the at least one second position. A feature recognition area of a target face image, wherein the image recognition unit calculates a pixel change value of the feature recognition area of the at least one target face image within a predetermined time, and matches the calculated corresponding to at least one target face The pixel change value of the image is used as the characteristic image change value of the at least one target facial image. A video conference management method suitable for a video conference in a conference space, wherein the conference space has at least one participant, the method includes: receiving a sound emitted in the conference space; according to the received Using the sound to determine a first position of the sound; capturing a panoramic image of the conference space; identifying a facial image of at least one participant in the panoramic image, and determining a second position of the at least one facial image; Determine a speaker of the at least one participant according to the first position, the at least one second position, and the at least one facial image; and set the panoramic image to be displayed in a first area of a video conference image and zoom in The image of the speaker in the panoramic image is determined, and the enlarged image of the speaker is set to be displayed in a second area of the video conference image. The video conference management method according to item 6 of the scope of patent application, further comprising: performing a voice-to-text operation on the voice issued by the speaker to convert the voice of the speaker into a text corresponding to the speaker Message, and store an identifying name corresponding to the speaker and the text message to a conference record database. The video conference management method according to item 6 of the scope of patent application, wherein the step of determining the speaker among the at least one participant according to the first position, the at least one second position, and the at least one face image The method includes: setting a target area in the panoramic image according to the first position, and identifying at least one target facial image in the target area based on the target area and the at least one second position; and according to the at least one target The image change of the facial image determines the speaker among the at least one target participant to which the at least one target facial image belongs. The video conference management method according to item 8 of the scope of patent application, wherein the speaker is determined among the at least one target participant to which the at least one target facial image belongs based on the image change of the at least one target facial image. The steps include: calculating a characteristic image change value of the at least one target facial image; and if a maximum characteristic image change value of the at least one characteristic image change value exceeds a predetermined threshold, determining a meeting corresponding to the maximum characteristic image change value The speaker is the speaker of the at least one target participant. The video conference management method according to item 9 of the scope of patent application, wherein the step of calculating the characteristic image change value of the at least one target face image includes: according to the at least one second position corresponding to at least one target face Set the feature recognition area of the at least one target facial image with at least one target position of the partial image; and calculate a pixel change value of the feature recognition area of the at least one target facial image within a predetermined time, and calculate the calculated The pixel change value corresponding to the at least one target facial image is used as the feature image change value of the at least one target facial image.