TW202327343A - Video conference device and operation method thereof - Google Patents

Abstract

A conference device includes an image capturing unit, a communication unit and a processor. The communication unit communicatively connects to a server and a first electronic device located in the same space with the conference device. The processor is configured to perform an online conference program and process an audio-visual signal received from the server via the communication unit. The processor is configured to transmit certification information of the first electronic device to obtain a cursor operation authority. The processor is configured to receive a real-time visual signal comprising a real-time state generated by applying an operation instruction message matching the cursor operation authority to a representation cursor of the certification information and comprising a real-time state generated by applying the operation instruction message to a representation cursor represented a remote participant of the online conference.

Description

Video conferencing device and its operating method

This disclosure document relates to a smart conference system and its operating method, especially a smart conference system that allows multiple users to operate simultaneously and its operating method.

In a general online video conferencing system, a speaker usually displays his screen image or application program image, or the speaker chooses which participant's image to display. In this way, all participants can watch the same briefing screen to facilitate the discussion of the meeting. If it is necessary to discuss the content of the presentation during the meeting, only the screen provider can use the cursor to mark the position in the presentation screen, and other participants need to use language or body movements to express what he/she wants to discuss about the content of the screen. For example, if a participant asks a question somewhere on the presentation screen, others need to find out where the corresponding content is located on the projection screen based on the questioner's narration, and the cursor operator also needs to find out where the corresponding content is located on the screen. The position can be moved to mark the cursor, which will cause a lot of trouble for large online meetings. Participants in the meeting may not be able to immediately know which part of the presentation screen the content of the current discussion is displayed on, resulting in the need to wait for everyone to understand before continuing the meeting, or there may be participants who cannot follow up at any time Inconvenient situation discussed at the meeting.

The disclosure document provides a video conferencing device including an image capture component, a communication component and a processor. The communication component is communicatively coupled to the server and a first electronic device located in the same space as the video conferencing device. The processor is used for executing the online meeting program and processing the audio-visual signal received from the server through the communication element. Wherein the processor executes the online conference program to join the online video conference in the server, the processor obtains the first authentication information from the first electronic device through the communication element, transmits the first authentication information for identity authentication, and controls the server to provide the first electronic device Corresponding to the first cursor operation authority of the first authentication information, the first cursor operation authority is used to set the first representative cursor of the first identity information in the online video conference. The processor collects the first operation instruction message from the first electronic device through the communication element, processes the first operation instruction message in real time, and sends the processed first operation instruction message to the server. The communication component receives the real-time video signal of the online video conference from the server, the processor processes the real-time video signal and outputs it to be displayed on the display component, and the real-time video signal includes the first operation instruction message in compliance with the first cursor operation authority to act on the first representative The first generated by the cursor represents the real-time state of the cursor or the second generated by the second represented by the cursor acting on the remote participant in the online video conference represents the real-time state of the cursor.

The disclosure document provides a video conferencing device including an image capture component, a communication component and a processor. The image capture component is used to capture the physiological characteristics of the first user located in the same space as the video conferencing device as authentication information. The communication component is used for communicating with the server. The processor is used for executing the online meeting program and processing the audio-visual signal received from the server through the communication element. The processor executes the online conference program to join the online video conference on the server, the processor receives the physiological characteristics of the first user through the image capture device, and sends the authentication information through the communication device for identity authentication, so as to control the server to provide the first user The user corresponds to the cursor operation authority of the authentication information, and the cursor operation authority is used to set the first representative cursor of the identity information in the online video conference. The processor receives the specific gesture of the first user through the image capture device, and recognizes the specific gesture to obtain the first operation instruction message, wherein the first operation instruction message includes an operation instruction for the first representative cursor, and the processor processes the first one in real time. Operating the instruction message and sending the processed first operation instruction message to the server. The communication component receives the real-time video signal of the online video conference from the server, the processor processes the real-time video signal and outputs it to be displayed on the display component, and the real-time video signal includes the first operation instruction message that meets the cursor operation authority and acts on the first representative cursor. The generated first represents the instant state of the cursor or acts on the second representative cursor representing the remote participant in the online video conference. The generated second represents the instant state of the cursor.

This disclosure document provides a method for operating a video conferencing device, including the following operations: joining an online video conference through a server; obtaining authentication information of a user or electronic device located in the same space as the video conferencing device, wherein the authentication information is extracted from an image Receive components or communication components; transmit authentication information for identity authentication; control the server to provide user or electronic device authentication information cursor operation authority in the online video conference, where the cursor operation authority is used to set the authentication information in the online video conference The first representative cursor; collect and process the first operation instruction message from the user or the electronic device, wherein the first operation instruction message includes an operation instruction to the first representative cursor, and the processor processes the first operation instruction message in real time and processes it The first operation instruction message is sent to the server; the real-time image signal is received from the server through the communication component, and the real-time image signal includes the first operation instruction message that meets the cursor operation authority and acts on the first representative cursor generated by the first representative cursor. The real-time status or the real-time status of the second representative cursor generated by the second representative cursor representing the remote participant in the online video conference; the real-time image signal is processed and output to be displayed on the display unit.

One of the advantages of the above-mentioned smart conference system and operation method is that it is convenient for participants other than the speaker to explain and illustrate.

Embodiments of the disclosed document will be described below in conjunction with related figures. In the drawings, the same reference numerals indicate the same or similar elements or method flows.

FIG. 1A is a schematic diagram illustrating the operation of a plurality of smart conference systems 110 a and 110 b according to an embodiment of the disclosure. The smart meeting systems 110a and 110b are used to conduct an online meeting through a server, and the online meeting allows remote users 150c and 150d to join through remote mobile devices 160c and 160d respectively. In some embodiments, the smart conference system 110a, the smart conference system 110b, the remote user 150c, and the remote user 150d may not be in the same space, and communicate with each other through the network. For example, the smart conference system 110a, the smart conference system 110b, and the remote mobile devices 160c and 160d respectively owned by the remote users 150c and 150d are coupled to each other via a network.

The smart conference system 110a includes a conference device 120a, a display device 140a, and multiple mobile devices 160a located near the conference device 120a (for example, in the same conference room). The smart conference system 110b includes a conference device 120b, a display device 140b, and a plurality of mobile devices 160b located near the conference device 120b (for example, in the same conference room). The conference devices 120a and 120b are used to connect to the server to establish or participate in an online conference, and are used to control the display images of the display devices 140a and 140b according to the data sent by the server. For the convenience of explanation, this disclosure document describes the conference device 120a of the smart conference system 110a as the host device (Host) of the online conference, and describes the conference device 120b of the smart conference system 110b as the conference participant device (Participant) of the online conference, However, this disclosed document is not limited thereto, and the conference devices 120a and 120b of the smart conference systems 110a and 110b can also exchange identities with each other. In this disclosure document, the smart conference systems 110a and 110b or the devices used by other users have the function of creating an online conference on the server and obtaining the authority of the conference host device. The conference host device has all the settings for the online conference it hosts. Participant operation authority and other functions, and conference participating devices can join the created online conference. The conference participating devices are not limited to the number shown in the figure. Figure 1A is only used for illustration, and the relevant content will be explained in detail in subsequent paragraphs.

Please refer to the online conference system 110a in FIG. 1A again, the mobile device 160a is used to communicate and couple to the conference device 120a in a wired or wireless manner. The user 150a can log in to the online meeting through the conference device 120a to obtain operations on the online meeting by operating the online meeting program of the mobile device 160a or through feature recognition (such as face recognition, voiceprint recognition, gesture recognition, or voice recognition). authority, and send an operation command message to the conference device 120a. In some embodiments, if the user 150a carries the mobile device 160a and moves to a space different from that of the smart conference system 110a, the user 150a can be regarded as the remote user 150c or 150d, and the mobile device 160a it carries can be viewed as It is the remote mobile device 160c or 160d. The remote mobile device 160c or 160d can directly connect to the server through the online conference program without going through the smart conference system 110a or 110b. The aforementioned descriptions about the online meeting system 110a are also applicable to the online meeting system 110b, and for the sake of brevity, details are not repeated here.

In some embodiments, if the remote mobile devices 160c and 160d join the online conference established by the conference device 120a of the smart conference system 110a, the remote mobile devices 160c and 160d directly obtain the operation authority from the server, and can directly send the operation command message to the server. The details of how the remote mobile devices 160c and 160d obtain the operation authority and transmit the operation instruction message will be described in the following paragraphs, and will not be repeated here. In some embodiments, there may be more than two smart conference systems participating in the online conference at the same time, and the conference device (such as the conference device 120a of the smart conference system 110a) that establishes the online conference is used as the conference host device, and the remaining conference devices (such as The conference devices 120b) of the smart conference system 110b are all conference participating devices.

In some embodiments, the user 150a of the smart conference system 110a can establish an online conference on the server through the conference device 120a, and establish multiple types of operation rights (such as senior executives and customers) and set various types through the conference device 120a The authority scope of the operation authority.

FIG. 1B is a simplified functional block diagram of the conference device 120a in FIG. 1A. The conference device 120a includes an image capture component 130a, a processor 170, a communication component 180, and an audio capture component 190 (such as a microphone). The processor 170 is coupled to the communication component 180, executes the online conference program and provides user interface and interface functions for connecting devices. The communication component 180 is used to communicate with the mobile device 160a and the server, wherein the communication component 180 can be connected through a signal cable (such as an HDMI transmission cable), a network cable, Bluetooth, Wi-Fi direct connection or other suitable wired or wireless connection methods. Realize the communication connection with the mobile device 160a, and connect to the server through the Internet. The processor 170 is also coupled to the image capture unit 130a, the audio capture unit 190 and the display device 140a in FIG. 1A. In some embodiments, the processor 170 is also communicatively connected to the image capture unit 130b. The image capture component 130a is used for acquiring facial feature and/or gesture information of conference participants in the conference room. The audio capture component 190 is used for acquiring voice information of conference participants. The processor 170 executes the operating system and the online meeting program, and is used to obtain the operation command message from the information collected by the image capture unit 130a and the audio capture unit 190 and send it to the server, and is used to transmit the audio and/or information from the server. Or the video data controls the display device 140a to display the corresponding screen content (such as a conference briefing). In addition, the execution of the online meeting program by the processor 170 can also be used to establish an online meeting on the server, and to establish multiple types of operation authority on the server and set the authority range of each type of operation authority.

In some embodiments, the image capturing component 130a may be implemented by a video camera, a digital camera, a camera unit on a mobile device, or other suitable electronic devices capable of capturing image frames. In some embodiments, the display device 140a may be implemented by a flat panel display, a projector or other suitable electronic devices capable of displaying images. In some embodiments, the display device 140a can be integrated with the conference device 120a. In some embodiments, the processor 170 may be implemented by a single or multi-chip general-purpose processor, an application specific integrated device (ASIC) or a field programmable gate array (FPGA). In some embodiments, the mobile device may be a smart phone, a tablet computer, a notebook computer or other suitable mobile devices.

The foregoing connections, implementations, and advantages of various components in the conference device 120a are also applicable to corresponding components in the conference device 120b, and for the sake of brevity, details are not repeated here.

The following uses FIG. 1A as an example to illustrate the operation process of the smart conference system disclosed in this document. The user 150a uses the online meeting program on the mobile device 160a, computer or conference device 120a to log in to the video conference server to establish a video conference. The server will give the user 150a the status of the conference moderator, and the user 150a can select other users and send out the conference Invitation, the server generates a meeting invitation notification and sends it to the contact mailbox or smart meeting system account of the selected user. The user 150a gives corresponding conference operation authority according to the identity category of the participants. The identity categories from high to low may include moderator, speaker, general participants and observers. The identity category can also be other users and corresponds to the company or organizational level of the user 150a, including senior executives, department heads, employees, and visitors from high to low. It is set by the host according to the identity category. The conference operation authority includes whether to display the cursor movement, coloring mark, underline mark, text annotation, page change, page zoom, rotation angle, and new sharing of the participants of the identity. For operations such as screen windows and displaying new content screens, the user 150a can also set whether a participant with a higher identity category (that is, a higher operation authority) will not display a lower identity (that is, a lower operation authority) when speaking or operating. Participant actions. Then other users 150b, remote users 150c and 150d use the mobile device 160b, the remote mobile device 160c (such as a computer), the remote mobile device 160d or the online conference program on the conference devices 120a and 120b to log in to the video conference server, Register to participate in the meeting through the meeting invitation notice, the server will provide the information of the meeting participants to the meeting host user 150a, the meeting host user 150a can approve or reject the meeting participants to participate in the meeting, and set the meeting identity of the participating participants as Moderator, presenter, regular attendee or observer visitor. When the video conference starts, the host user 150a can choose to share images, files or files uploaded by any user through the online conference program to all devices connected to the server (such as conference devices 120a and 120b, mobile device 160a and 160b and remote mobile devices 160c and 160d), the uploaded images, files or files are received by the server, processed to generate shared images and then transmitted to all user terminals such as all devices connected to the server, so all users Participants can see the sharing screen selected by the host through the online meeting program. In the video conference, the users 150a and 150b, the remote users 150c and 150d control the cursor or perform operations on the online conference program on the mobile devices 160a and 160b (such as computers), the remote mobile devices 160c and 160d, and the conference device 120a or 120b. Other functional operations generate operation instruction messages. The operation instruction messages of the users 150a and 150b will be collected by the conference devices 120a and 120b respectively and sent to the server. The operation instruction messages of the remote users 150c and 150d will be sent to the remote mobile devices 160c and 160d are directly transmitted to the server, and these operation instruction messages can be processed in real time and the server corresponds to each user's authority and cursor. If a user's operation instruction message does not meet its authority, it will be regarded as For invalid operations, all valid operations sent by users at the same time will be processed together, and the server will process the valid operations and received upload screens to generate a sharing screen, and the sharing screen will immediately display the operation results such as cursor movement that meets the permissions , so that the user in the meeting can see other people's effective operations in the sharing screen of the online meeting program. In some embodiments, the user 150a or 150b, the remote user 150c or 150d can respectively use the mobile device 160a, 160b, the remote mobile device 160c, 160d or the conference device 120a or 120b to establish an online conference at any time through the installed online conference program. Meetings, invite participants to join or join established meetings, upload shared images, issue operating instructions, and synchronize the changes caused by the operating instructions issued by participants in the shared images to conduct video conferences.

FIG. 2 is a flowchart of an operating method 200 according to an embodiment of the disclosure. The operation method 200 is applicable to the conference device 120a in FIG. 1A or other devices installed with an online conference program, and only the conference device 120a is used as an example for illustration. A plurality of conference devices each executing an online conference program can cooperate to hold an online conference through a server. The process of executing the operation method 200 by an individual conference device (taking the conference device 120a as an example) will be described below.

In step S210, the processor 170 acquires a plurality of authentication information from a plurality of instruction sources through the communication component 180 or the image capture component 130a. In some embodiments, multiple command sources may include multiple mobile devices 160a. When an unspecified mobile device 160a enters the communication range of the communication element 180, the user 150a can open a specific application program or connect to a specific web page through the mobile device 160a to connect to the server through the conference device 120a, and then input and send the personal authentication information to log into the video conference. In some embodiments, the personal authentication information may be one or more of the following: (1) facial features of the user obtained through the image capture component 130a; and (2) identification information of the user obtained through the communication component 180 , wherein the identification information includes one or more of the following: e-mail address, combination of account number and password, phone number, biometric data, and International Mobile Device Identification Number, etc. The processor 170 receives the authentication information from the mobile device 160a through the communication element 180, so as to determine the respective operation rights of the multiple mobile devices 160a of the multiple users 150a in the subsequent step S220.

In some embodiments, multiple instruction sources may include multiple users 150a. When an unspecified user 150a enters the field of view of the image capture component 130a, the processor 170 will capture the image of the user 150a through the image capture component 130a. The facial features are used as the authentication information to determine the respective operation rights of the multiple users 150a in the subsequent step S220.

In step S220, the processor 170 transmits the authentication information obtained in step S210 to the server through the communication element 180 for identity authentication, so that multiple instruction sources respectively obtain multiple operation rights to control a shared content of the online meeting. In some embodiments, the shared content may be an application (such as text editing software or presentation software) screen shared by a certain mobile device 160a or 160b of the smart meeting system 110a or 110b to all participants in the online meeting, or the remote mobile device 160c Or 160d to share the application screen of all participants in the online meeting, or the screen of the cloud application stored in the server, the host device can choose to share with all participants in the online meeting at any time or set the screen sharing permissions of the participants, allowing Participants upload and share screens during the meeting.

In some embodiments, identity authentication includes comparing the authentication information with the registration information of a user stored in the server, where the registration information may include one or more of the following: personal email address, combination of account number and password, international Mobile device IDs, phone numbers, biometrics, and more. In some embodiments, the registration information also includes the user's class information, where the class information can be distinguished by company ranks, such as senior executives, general employees or customers, and so on. In the server, the registration information of the users of each company or organization is set as the same group, and the authenticated group manager performs identity verification and management. After the identity authentication is successful (that is, the authentication information matches the registration information), the server will give the corresponding instruction source and the corresponding operation authority corresponding to the class information in the registration information according to the setting of the conference creator.

In some embodiments, the meeting creator can create a registration program and select relevant users before the meeting starts, and the server will generate a registration link, page or notification to the relevant users, and the relevant users can use the registration link, page or online The meeting program joins the meeting and becomes a participant. The meeting creator can set the operation rights of the participants in the video conference, and set whether irrelevant users can attend the meeting, whether they can share the screen or discuss aloud, and whether to display the corresponding cursor. In some embodiments, users who want to participate in the meeting can use a mobile device or computer to execute an online meeting program to sign up for the meeting. The user provides an account, authorizes to read the user identity information of the company or organization system to which the user belongs, or The user provides the identity information himself, and the meeting host can set the requirement for the user to perform identity authentication, such as receiving the authentication letter or authentication code through the user 150a's company mailbox or mobile device 160a, and the user 150a will reply to the smart conference system 110a for authentication .

In some embodiments, when multiple users from different groups hold an online meeting, the meeting host can operate the online meeting program to read from the server the verified identity information of the participating users in their group, such as company name In the video conference screen, the identity information can be marked next to the user name, avatar, the user's video screen or the cursor corresponding to the user. In some embodiments, the user's identity information is stored in the database or smart conference system managed by the company or organization group to which the user belongs. When the user hosts or participates in a video conference, the server receives the The database or smart conference system reads the user's identity information, and decides which permissions to grant and whether to mark the user's identity information in the video conference according to the host's settings.

In some embodiments, if the server judges that the source of the instruction is a senior supervisor after identity authentication, the server will grant the following operation permissions: the permission to switch the page of the shared content, the permission to zoom in or out the page of the shared content, The authority to move the cursor belonging to the source of the instruction, the authority to grant or remove part or all of the operation authority of other users, the authority to set whether to display the cursor corresponding to a certain or all other users, and the authority to set other users except for moving the cursor Whether you can perform other operations in the shared content, etc. For another example, if the processor 170 authenticates the identity of the instruction source as a client, the processor 170 may set its operation authority to include only the authority to move the cursor belonging to the instruction source. In some embodiments, the operation authority of the conference participants may be higher than that of the conference moderator.

In step S230, the processor 170 determines whether an operation instruction message is received from the instruction source. In some embodiments, the operation instruction message includes an instruction to perform one or more of the following operations: moving the cursor in the shared content, switching pages of the shared content (such as switching slideshow pages or switching desktops), zooming in and out of objects in the shared content, Operations such as granting or removing user operation permissions. If the processor 170 does not receive the operation command message, the processor 170 may repeatedly execute step S230 until the operation command message is received. On the other hand, if the processor 170 determines that the operation instruction message has been received, the processor 170 may then execute step S240.

In some embodiments, before step S230, the processor 170 may receive a trigger control message from the user (for example, the user poses a predetermined gesture such as lifting a finger) from the image capture component 130a (and 130b) or the audio capture component 190 hand and straight arm, or the user speaks a predetermined phrase), or the communication element 180 receives a trigger control message from the mobile device 160a (or 160b) (for example, the user picks up the mobile device connected to the video conference and shake up and down, or move the cursor or click the displayed function menu, option or button in the video conference). The processor 170 may proceed to step S230 to determine whether an operation command message is received from the command source after receiving the trigger control message, so as to save computing resources of the processor 170 . In some embodiments, the same authentication information may correspond to a user 150a (or 150b) and its own mobile device 160a (or 160b), and the server may grant the operation authority to the user 150a and its own in the process S220 at the same time. mobile device 160a, but at the same time, the processor 170 may only receive the operation instruction message sent by one of the user 150a (or 150b) and its mobile device 160a (or 160b). That is, if the user 150a (or 150b ) simultaneously uses gestures and the mobile device 160a (or 160b ) sends multiple operation command messages, the processor 170 will choose one to process.

In step S240, the processor 170 collects the operation instruction messages sent simultaneously from all instruction sources in real time, and transmits these operation instruction messages to the server together. In this way, multiple users 150a can operate the shared content of the online conference through the conference device 120a together. Furthermore, multiple users 150a in the smart conference system 110a and multiple users 150b in the smart conference system 110b can operate the shared content of the same online conference through their respective conference devices 120a, and the smart conference system 110a and the The smart conference systems 110b will respectively collect the operation command messages sent at the same time in real time, and send them to the server for real-time processing, and display the operation results on the video conference screen at the same time.

In some embodiments, the server screens the operation instruction messages of each instruction source according to the operation authority given to each instruction source in step S220. For example, a certain user 150a is authenticated as a client in step S220, and the conference moderator sets the screen operation authority of the client identity to only move the cursor in the shared screen, then the server can respond to the user 150a with gestures in subsequent steps Or the command message of moving the cursor operation issued by voice, and ignore the command message of switching page operation issued by the user.

In some embodiments, the server sets each instruction source to prohibit it from generating one or more types of operation instruction messages according to the operation authority given to each instruction source in step S220. For example, the mobile device 160a of a certain user 150a is authenticated as a client in step S220, and the conference host sets the screen operation authority of the client identity to only move the cursor in the shared screen, and the server will prohibit the user in step S220. The mobile device 160a generates the following types of operation message commands through the application program: multiple operation message commands such as switching pages, zooming in and out, etc., and only allows the application program of the mobile device 160a to generate operation command messages for moving its own cursor. The remote mobile device 160c After receiving the types of prohibited operation message commands, the application will lock the permissions and related settings that generate these types of operation message commands during the video conference, and unlock them after leaving the video conference. The server will not in the next steps Respond to the operation message command of the prohibited category.

In some embodiments, the conference device 120a may also filter the operation instruction messages of each nearby instruction source according to the operation authority granted to each nearby instruction source in step S220, and then transmit the filtered operation instruction messages to the server. In some other embodiments, the conference device 120a may also set each nearby instruction source to prohibit it from generating one or more types of operation instruction messages according to the operation authority given to each nearby instruction source in step S220.

In some embodiments, the user 150a can generate an operation command message to the conference device 120a through the mobile device 160a, gesture or voice. In order to obtain the operation instruction information, the processor 170 can (1) receive a specific gesture from the instruction source (such as the user 150a) through the image capture component and obtain the corresponding operation instruction information through the image recognition technology, (2) through the audio capture component Receive a voice message from an instruction source (such as user 150a) and obtain a corresponding operation instruction message through voice recognition technology, and/or (3) receive a specific operation (such as rotating ) and parse the content of the communication packet to obtain the corresponding operation instruction message. The processor 170 will determine whether the received operation command message is from the user 150a (such as gesture or voice), or from the mobile device 160a (such as a message packet generated by a mobile phone), and obtain the operation issued by the user accordingly. instruction. In some embodiments, the processor 170 processes operation command messages from different command sources, converts them into message packets of the same format and transmits them to the server, and the server performs message source judgment and subsequent processing.

In some embodiments, the remote mobile devices 160c and 160d also include image capture components and audio capture components. The remote mobile devices 160c and 160d can obtain the facial features or voiceprints of the remote users 150c and 150d respectively, and send the facial features or voiceprints as authentication information to the server for the server to determine the remote users 150c and 150d respectively, or remote mobile devices 160c and 160d respectively. In this way, the remote users 150c and 150d can also use voice, gestures and/or specific operations on the remote mobile devices 160c and 160d to send operation command messages to the server, for example, when making a video call to the mobile device, at the same time An operation command message is sent through the image capture component or the audio capture component to operate the presentation screen of the video conference.

In step S250, the processor 170 receives from the server the content of the shared screen that is immediately processed according to the received operation instruction. For example, if the processor sends the page switching operation command message and the cursor moving command message from the two users 150a respectively to the server in the process S240, the server will return in the process S250 that the two operation commands have been used. The adjusted shared content (that is, the cursor has been moved and the page has been switched) is sent to all participating devices. After the communication component 180 receives the new shared screen content, the processor 170 will control the display devices 140a and 140b to display the shared screen content. In a word, by executing the operation method 200, the online conference system and the conference device 120a allow multiple users 150a and 150b, multiple remote users 150c and 150d in the same or different spaces to simultaneously operate the shared screen content of the online conference, and automatically Different participating devices watch the content of the shared screen after synchronous processing, so that each user 150a, 150b, remote user 150c and 150d can directly point out the part they want to discuss in the shared screen of the online meeting, so that other users Participants can immediately notice the key points of the discussion, making the discussion process of the online meeting as easy to understand as a live discussion. The details of step S250 will be further described in the subsequent paragraphs in conjunction with FIG. 3 .

FIG. 3 is a detailed flowchart of step S250 in FIG. 2 of this disclosure document. As mentioned above, in video conferencing, the operation instruction messages generated by different instruction sources may contain various operation instructions. After screening and processing according to the operation authority, the content of the shared screen will change. In order to simplify the description, the The process only describes two instructions of moving the cursor and switching pages, and the steps corresponding to other types of instructions (such as coloring, marking, adding annotations, zooming in and out objects or switching desktops, etc.) can be arranged before step S310, step Between S320 and S330 or after step S340.

In step S310, the server or the processor 170 determines whether the operation instruction message from the instruction source includes an instruction to move the cursor. If the operation instruction message has an instruction to move the cursor, the processor 170 may proceed to step S320. If the operation instruction message does not include an instruction to move the cursor, the processor 170 may skip step S320 and directly enter step S330.

In step S320, the server or processor 170 has judged in step S310 that the operation instruction message includes an operation instruction that the user 150a intends to move the cursor, and the processor 170 will further move the cursor according to the content of the operation instruction message, for example, according to the content of the operation instruction message Coordinates, movement vectors, or other parameters move the cursor to a specific location on the shared screen. In some embodiments, the user 150a sends out other operation instructions besides moving the cursor, for example, in an online meeting program on a mobile device, selects to add a mark, an annotation or other functions, and the operation instruction message will be added in addition to the operation instruction for moving the cursor. The server will process the operation instructions corresponding to these functions, and display the colored mark, crossed mark or other possible marking results based on the cursor movement track in the shared screen content of the video conference, or the cursor in the shared screen content The comment field is displayed at the location, and the user can then operate the mobile device to input comments. The comment field displayed in the content of the sharing screen will display the user's input results in real time. The user can also choose to add pictures, tables, files, Links or text and so on. In one embodiment, the page of the shared screen content with markings, annotations or other content will be stored on the server side, and the shared screen content will be stored for a certain period of time after being switched to other pages or closed during the online meeting, so when sharing When the screen content is switched back to this page, the previously added marks, annotations or other content will still be displayed to all participants.

In one embodiment, the command to move the cursor comes from the user 150a pointing to the screen in the air to move up, down, left, and right with a gesture of raising a finger (such as an index finger), and the image capture component 130a will capture continuous images of the gesture movement , and the processor 170 will analyze the direction and vector of the gesture movement in the captured image, and generate an operation command for the cursor to follow the gesture movement of the user 150a, and transmit it to the server to generate the shared screen content after the cursor movement. In another embodiment, the command to move the cursor comes from specific words such as "move down" or "move to the right" spoken by the user 150a, and the audio capture component 190 will capture the voice audio sent by the user , the processor 170 will analyze and process the voice audio to obtain the voice operation command spoken by the user and upload it to the server, or upload the voice audio to the server and the server will analyze and process it to obtain the voice operation command, and the server will follow the instructions of the voice operation command The content of the shared screen after the cursor is moved is generated, wherein the cursor moves in a corresponding direction according to the voice command of the user 150a. In one embodiment, the user 150a can use voice or gestures to issue marking, annotation or other functional operation instructions, combined with the operation command of moving the cursor, to add marking, annotation or other content to the content of the shared screen, so that all online meetings Participants can see.

In another embodiment, please refer to FIG. 3 and FIG. 4 at the same time. The source of the command is the mobile device 160a. When the user rotates the mobile device 160a, the mobile device 160a and the online meeting program executed by it will follow three orientation axes (such as the x-axis, y-axis and z-axis in FIG. 4 ), generate communication packets and transmit them to the nearby conference device 120a, and the processor 170 can analyze these communication packets to obtain operation instructions message, and the operation command message includes a corresponding cursor move command. For example, if the communication packet contains the angle of rotation of the mobile device 160a based on the x-axis, the processor 170 will obtain an operation command message with information about the vertical movement of the cursor, wherein the vertical movement information is used to control the vertical movement of the cursor in the content of the shared screen. Movement: The communication packet may also include the angle of rotation of the mobile device 160a with the z-axis as the axis, and the processor 170 will obtain an operation instruction message with horizontal movement information, wherein the horizontal movement information is used to control the level of the cursor in the content of the shared screen Movement: the user can use the mobile device 160a to rotate freely, and the gyroscope, accelerometer or other sensors in the mobile device 160a will simultaneously detect the rotation angles with the x-axis and the z-axis as the axis, and obtain information corresponding to these rotations The angle operation command message makes the cursor move obliquely on the screen synchronously. In other words, the shared screen includes the real-time status of the cursor after the operation instruction message acts on the cursor, such as speed, acceleration, displacement, displacement direction and so on. In some embodiments, the rotation of the mobile device 160a around the y-axis will trigger the processor or the server to determine whether the mobile device 160a has the ability to adjust the size of the content page in the sharing screen, rotate the angle of the content page in the sharing screen, or Screen switching content page permissions, if the mobile device 160a has permission, then according to the direction and angle of the mobile device 160a's rotation around the y-axis, adjust the content page size in the sharing screen, rotate the content page, or switch the content page on the sharing screen The server will generate and share screen content to the participating devices based on the operation message command. It is worth noting that the commands generated by the mobile device 160a rotating on three axes can be set and changed according to the needs of the user 150a, and can be replaced with each other, and are not limited to the above exemplary embodiments.

In some embodiments, the x-axis and y-axis are coordinate axes on a horizontal plane perpendicular to the direction of gravity, and they are perpendicular to each other, and the z-axis is a coordinate axis perpendicular to the horizontal plane and parallel to the direction of gravity.

In step S330, the server or the processor 170 determines whether the operation instruction message from the instruction source includes an instruction to switch pages. If there is an instruction to switch pages, the processor 170 will enter step S340. If there is no instruction to switch pages, the processor 170 may skip step S340 and execute step S230 again. In some embodiments, in step S330, the server or the processor 170 also determines whether the operation command message includes adjusting page zoom, rotating angle, capturing images, adding a new sharing screen window, and displaying a file or information selected by the user on the In the window of the sharing screen, ... and other functional commands are screened and executed according to the permission settings.

In step S340, the server or processor 170 has judged in step S330 that the operation instruction message includes the operation instruction that the user 150a wants to switch pages. In one embodiment, the instruction to switch pages comes from the user 150a, and the user 150a can The page is switched by moving left and right in the air through the gesture of straight five fingers. For example, when the gesture moves to the left and right (or up and down) of the user 150a respectively, the processor 170 can use image recognition technology to share The slides in the content are switched to the previous page and the next page respectively, but the disclosed document is not limited thereto. In another embodiment, for example, when the gesture moves to the left and right (or up and down) of the user 150a respectively, the processor 170 may use image recognition technology to move the desktop of the operating system in the shared content to the left. Move right and left to switch between multiple operating system desktops. In another embodiment, the command to switch pages comes from specific words such as "next page" or "previous page" spoken by the user 150a, and the processor 170 will use the voice recognition technology according to the user's voice command. Switch the slideshow in the shared content to the corresponding page In some embodiments, please refer to FIG. 3 and FIG. 4 at the same time. The instruction source is the mobile device 160a. When the mobile device 160a rotates in a specific direction, the processor 170 will be triggered to switch pages. For example, when the mobile device 160a rotates along the first direction on the y-axis (such as the rotation direction indicated by the arrow on the y-axis), the processor 170 will switch the slide page to the next page; when the mobile device 160a is on the y-axis The processor 170 will switch the slide to the previous page when rotating along the second direction (for example, the direction opposite to the rotation direction indicated by the arrow on the y-axis), but the disclosure is not limited thereto.

The method and steps described above in conjunction with FIG. 3 and FIG. 4 for the user 150a and the mobile device 160a to generate an operation command message are also correspondingly applicable to the remote users 150c and 150d and the remote mobile devices 160c and 160d. In step S330, the server may process and determine the authority of the operation commands generated by the remote users 150c and 150d and the remote mobile devices 160a and 160b. For the sake of brevity, details are not repeated here. In some embodiments, the remote users 150c and 150d can use the touch screen, mouse, keyboard, trackball or other input devices of the remote mobile devices 160c and 160d to operate, and the remote mobile devices 160c and 160d will generate Corresponding operation command messages are screened and processed by authority settings, and receive real-time updated shared screen content from the server, showing the changes produced by remote users 150c, 150d or other online conference participants after operations, such as cursors Move, color mark, line mark, text annotation, page change, page zoom, rotation angle, snapshot, add a new sharing screen window and display a new content screen, etc.

FIG. 5 is a schematic diagram of shared content 500 displayed by the display device 140a or 140b according to an embodiment of the disclosure. FIG. 5 is used to illustrate the situation where multiple users 150a and 150b, remote users 150c and 150d operate the shared content 500 at the same time. The cursor 510a is the cursor controlled by the user 150a, and the cursor 510b is the cursor controlled by the remote user 150d. In the figure, the identity of the user is marked next to each cursor. The moderator or the user can set whether to display the identity marked. In some embodiments, please refer to FIG. 1 and FIG. 5 at the same time. The cursor 510a can be understood as the cursor controlled by the user 150a using gestures, voice or the mobile device 160a, and the cursor 510b can be understood as the cursor used by the remote user 150d. Gesture, voice, or a cursor manipulated by the remote mobile device 160d. In some embodiments, as shown in FIG. 5, one or more cursors can be displayed in the shared content 500 at the same time and the movement of the cursors can be displayed in real time, and these cursors can be marked with different labels, colors, cursor patterns or names to identify each other. partition. In some embodiments, the processor 170 can control the cursors 510a and 520b respectively according to the needs of the user 150a and the remote user 150d (for example, operation instruction messages), for example, move the cursors 510a and 520b respectively according to the operation instructions, and according to the host Human or user settings correspond to showing or hiding cursors 510a and 520b. For example, when the user 150a is speaking, the processor 170 can display or hide the cursor 510b according to the setting of the meeting moderator or the user 150a, and hide the cursor 510a or other non-speaking cursors according to the setting of the meeting moderator. In some embodiments, when the cursors of other non-speakers are hidden, the non-speaking users can still see their own cursors on their mobile devices or on the display device 140a of the nearby smart conference system 110a, but The rest of the participants will not see the cursors of other non-speaking users on their respective mobile devices, and the display device of the remote smart conference system 110b will not display the cursors of non-speaking users either. The above only takes the cursor movement as an example. The smart conference system 110a of the present invention can also process other functional operation instructions of the user, including color marking, underline marking, text annotation, page change, page zoom, rotation angle, capture, etc. map, add a new share screen window and display a new content screen, etc. In one embodiment, the meeting host can set that the result of the operation command when the participant is not speaking will not be displayed to all participants, and only the user who has not spoken but has issued the operation command will be displayed on the display device of his mobile device or smart conference system You can see the operation results on the screen. When a user speaks, the previous operation results will be shared with other participants and displayed on the sharing screen, such as cursor position, colored mark, crossed mark, text annotation, Page changing, page zooming, rotation angle, adding a new sharing screen window, displaying a new content screen, etc., so that users can generate operation instructions in advance when they are not speaking, without occupying the meeting time when speaking.

In some embodiments, the meeting moderator of the smart conference system can also set: whether the cursor corresponding to each user is displayed at any time, whether the user who has no right to switch pages can request to switch pages, whether the observer or visitor user can speak, whether Automatically generate meeting records and provide to which users and so on. In some embodiments, the meeting host can obtain the meeting participants and their authenticated registration identity information. If there are participants or participants whose identities cannot be authenticated or registered without identity, the smart conference system will notify the meeting host. The meeting host can remove the participant through the online meeting program. In some embodiments, the meeting host can set the meeting participant as the speaker. The speaker can select a file or window screen on the mobile device or computer to upload and share the screen with other users, and can switch pages by operation. When the speaker speaks Or when operating functions such as cursor movement, the server will only display the operation results of the host and the speaker, and the cursor movement or other function operation results of other users will not be displayed on the sharing screen. In some embodiments, users who are not hosts or presenters can still see the cursor movement or function operation results sent by themselves on the video conference sharing screen of their own devices, but the server will only transfer the cursor movement or function operation results The screen is provided to the user's mobile device, and other users' devices will not receive the screen, that is, users who are not hosts or lecturers can only operate the cursor movement or other functional operations of their own devices.

In some embodiments, the meeting moderator can initiate voting during the meeting, and the moderator can set himself or other participants as the moderator by the conference founder, and the moderator can also set or select which participants can participate in the voting, such as identity Only users above a certain level can participate in voting. The online conference program provides a setting interface for the moderator to set the voting content and select the people who can vote. According to the moderator's setting, it provides a voting function interface for participants to decide their Voting options, the server will receive and count the voting options completed by the participants, and display the voting results on the screen of the online meeting program according to the settings of the moderator.

The smart conference system disclosed in this document allows multiple users to simultaneously control the displayed images, so that participants other than the speaker can explain or ask questions during the conference. Compared with the traditional conference system that requires the user to use a laser pointer or go to the console to explain, if it is in a different location, it is impossible to quickly explain and ask questions. The smart conference system disclosed in this document can facilitate participants other than the speaker Quickly perform operations and explanations, and through joint control of the smart conference system, it is convenient for users to discuss with each other.

Certain terms are used in the specification and claims to refer to particular elements. However, those skilled in the art should understand that the same element may be called by different terms. The description and the scope of the patent application do not use the difference in the name as the way to distinguish the components, but the difference in the function of the components as the basis for the distinction. The term "comprising" mentioned in the specification and scope of patent application is an open term, so it should be interpreted as "including but not limited to". In addition, "coupling" here includes any direct and indirect connection means. Therefore, if it is described that the first element is coupled to the second element, it means that the first element can be directly connected to the second element through electrical connection or signal connection means such as wireless transmission or optical transmission, or through other elements or connections. The means are indirectly electrically or signally connected to the second element.

The description of "and/or" used herein includes any combination of one or more of the listed items. In addition, unless otherwise specified in the specification, any singular term also includes plural meanings.

The above are only preferred embodiments of this disclosure document, and all equivalent changes and modifications made according to the requirements of this disclosure document shall fall within the scope of this disclosure document.

110a, 110b: Smart conference system 120a, 120b: conference device 130a, 130b: image capture components 140a, 140b: display device 150a, 150b: user 150c, 150d: remote users 160c, 160d: remote mobile devices 160a, 160b: mobile device 170: Processor 180: Communication components 190:Audio capture component 200: Operation method S210, S220, S230, S240, S250: steps S310,S320,S330,S340: steps x,y,z: axes 500: display screen 510a, 520b: Cursor

FIG. 1A is a schematic diagram illustrating the operation of multiple smart conference systems according to an embodiment of the disclosure. Fig. 1B is a simplified functional block diagram of the conference device in Fig. 1A. FIG. 2 is a flow chart of an operating method according to an embodiment of the disclosure. Fig. 3 is a detailed flowchart of some steps in Fig. 2. FIG. 4 is a schematic diagram of a communication device according to an embodiment of the disclosure. FIG. 5 is a schematic diagram of shared content displayed by a display device according to an embodiment of the disclosure.

110a, 110b: Smart conference system

120a, 120b: conference device

130a, 130b: image capture components

140a, 140b: display device

150a, 150b: user

150c, 150d: remote users

160c, 160d: remote mobile devices

160a, 160b: mobile devices

Claims (10)

A video conferencing device, comprising: an image capture element; a communication element for communicating with a server and a first electronic device located in the same space as the video conferencing device; and a processor for executing an online conference program and processing audio-visual signals received from the server through the communication element; Wherein the processor executes the online conference program to join an online video conference on the server, the processor obtains a first authentication information from the first electronic device through the communication element, and transmits the first authentication information for identity authentication, To control the server to provide the first electronic device with a first cursor operation authority corresponding to the first authentication information, the first cursor operation authority is used to set a first representative of the first authentication information in the online video conference cursor; Wherein the processor collects a first operation instruction message from the first electronic device through the communication element, the processor processes the first operation instruction message in real time, and sends the processed first operation instruction message to the server device; Wherein the communication element receives a real-time image signal of the online video conference from the server, the processor processes the real-time image signal and outputs it for display on a display element, the real-time image signal includes the first cursor operation authorization The first operation command message acts on the real-time state of the first representative cursor generated by the first representative cursor or on the second representative cursor generated by a second representative cursor representing a remote participant in the online video conference Instant status. The video conference device as described in claim 1, wherein the processor executes the online conference program to establish the online video conference on the server, when the video conference device receives the first authentication information, if the first electronic device The first authentication information corresponds to the moderator of the online video conference, and the first electronic device obtains a moderator authority. The video conferencing device as described in claim 2, wherein the host authority includes: when one of the multiple participants of the online video conference speaks in the online video conference, hiding a plurality of representative cursors in the real-time video signal part of it. The video conferencing device as described in claim 1, wherein the real-time image signal includes a screen signal of shared content, and the screen signal of the shared content shows the real-time status of the first representative cursor or the real-time status of the second representative cursor, The processor controls the server to provide the first electronic device with a first content operation authority of the online video conference corresponding to the first authentication information, and the first content operation authority includes one or more of the following: in the shared content The permission to switch between multiple pages and the permission to zoom in or zoom out the page of the shared content. The video conferencing device as described in claim 1, wherein the processor performs one or more of the following: receiving a physiological characteristic of a first user through the image capture element, and transmitting the physiological characteristic through the communication element for identity authentication; receiving a second authentication information corresponding to the first user through the communication element to control the server to provide a second cursor operation authority for the first user corresponding to the second authentication information; and receiving a specific gesture of the first user through the image capture element, and identifying the specific gesture to obtain a second operation instruction message; Wherein the second operation command message includes an operation command for moving a third representative cursor, and the second cursor operation authority is used to set the third representative cursor. The video conferencing device as described in Claim 1, wherein the processor receives a communication packet generated when the first electronic device rotates through the communication element, and analyzes the communication packet to obtain the first operation instruction message; Wherein, the first operation instruction message includes vertical movement information or horizontal movement information, and the processor generates the vertical movement information or the horizontal movement information according to the rotation angle of the first electronic device. The video conferencing device as described in claim 4, wherein when the processor processes the first operation instruction message in real time and transmits the processed first operation instruction message to the server, the processor is used for: filtering the first operation instruction message according to the first content operation authority and the first cursor operation authority corresponding to the first authentication information; and When the first operation command message conforms to the first content operation permission or the first cursor operation permission, the first operation command message is sent to the server through the communication element. A video conferencing device, comprising: an image capturing component, used to capture a physiological feature of a first user located in the same space as the video conferencing device as authentication information; a communication component for communicating with a server; and a processor for executing an online conference program and processing an audio-visual signal received from the server through the communication element; Wherein the processor executes the online conference program to join an online video conference on the server, the processor receives the physiological characteristics of the first user through the image capture component, and transmits the authentication information through the communication component to carry out identity authentication, to control the server to provide the first user with a cursor operation authority corresponding to the authentication information, and the cursor operation authority is used to set a first representative cursor of the identity information in the online video conference; Wherein the processor receives a specific gesture of the first user through the image capture device, and recognizes the specific gesture to obtain a first operation instruction message, wherein the first operation instruction message includes an operation on the first representative cursor instruction, the processor processes the first operation instruction message in real time and transmits the processed first operation instruction message to the server; Wherein the communication element receives a real-time image signal of the online video conference from the server, the processor processes the real-time image signal and outputs it to be displayed on a display element, the real-time image signal includes the first cursor operation authority The operation instruction message acts on the real-time state of the first representative cursor generated by the first representative cursor or acts on the real-time status of the second representative cursor generated by a second representative cursor representing a remote participant in the online video conference state. The video conferencing device as described in claim 8, wherein the real-time image signal includes a screen signal of shared content, and the processor obtains a content operation authority of the online video conference corresponding to the authentication information from the server, the content The operation authority includes one or more of the following: the authority to switch between multiple pages of the shared content and the authority to zoom in or out the page of the shared content. A method for operating a video conferencing device, comprising: Join an online video conference through a server; Obtaining an authentication information of a user or an electronic device located in the same space as the video conferencing device, wherein the authentication information is received from an image capturing element or a communication element; Send the authentication information for identity verification; Controlling the server to provide the user or the electronic device with a cursor operation authority of the authentication information in the online video conference, wherein the cursor operation authority is used to set a first representative cursor of the authentication information in the online video conference ; collecting and processing a first operation instruction message from the user or the electronic device, wherein the first operation instruction message includes an operation instruction to the first representative cursor, a processor processes the first operation instruction message in real time and sending the processed first operation instruction message to the server; receiving a real-time image signal from the server through the communication element, the real-time image signal including the first operation instruction message conforming to the cursor operation authority to act on the real-time state of the first representative cursor generated by the first representative cursor or acting on the real-time state of a second representative cursor generated by a second representative cursor representing a remote participant in the online video conference; and The real-time image signal is processed and output to be displayed on a display device.

Images