JP7046546B2 - Conference support system and conference support program - Google Patents

Description

The present invention relates to a technique for supporting a conference, and more particularly to a technique applicable to a conference support system and a conference support program for supporting a conference between a plurality of bases.

For example, a so-called video conference system, a whiteboard system, or the like is known as a mechanism for supporting a conference between a plurality of bases including a remote location by using IT technology. For example, in "Skype (registered trademark, the same applies hereinafter) for Business" (Non-Patent Document 1), online group meetings and messaging can be conducted by voice and video between multiple bases, and the contents of images and materials can be displayed. It also has the function of a shared digital whiteboard. In addition, "Microsoft Surface (registered trademark, the same applies hereinafter) Hub" (Non-Patent Document 2) is equipped with a camera, sensor, etc. on a large screen display and cooperates with an external information processing device such as a PC (Personal Computer). As a device capable of this, the above-mentioned online group meeting is realized.

Further, for example, in Japanese Patent Application Laid-Open No. 2006-50500 (Patent Document 1), a conference participant sends information on his / her remarks during the conference to the conference server device via a client device, and receives conference support. At the same time, it is described that the voice statement is converted into text data in real time while the proceedings are in progress by voice recognition of the client device or the conference server device.

In addition, various techniques for evaluating the quality of the conference have been studied, and for example, it has been proposed to acquire statistical information such as the number of remarks by the agenda of the conference attendees and the average affirmative rate. As a technique related to this, for example, Japanese Patent Application Laid-Open No. 2013-8114 (Patent Document 2) describes a speech quality evaluation value formed from the speech state of a speaker and a rotational posture of attendees other than the speaker (patent document 2). It is stated that the quality evaluation of the meeting is calculated based on the posture evaluation values formed from prone, supine, and upright immobility).

Japanese Unexamined Patent Publication No. 2006-50500 Japanese Unexamined Patent Publication No. 2013-8114

“Skype for Business-Collaborate with Everyone on Any Device”, [online], 2017, Skype and / or Microsoft, [Search August 22, 2017], Internet <URL: https://www.skype .com / ja / business / skype-for-business /> “Microsoft Surface Hub | Collaboration Device”, [online], 2017, Microsoft, [Search August 22, 2017], Internet <URL: https://www.microsoft.com/microsoft-surface-hub/en -jp ＞

According to the conventional technology, the operation of the meeting is streamlined, including the sharing of information and the automatic creation of minutes by converting the remarks into text, including the meeting between multiple bases, and the content and quality of the meeting are evaluated. It is said that it is possible to improve the efficiency of meetings. On the other hand, there are various viewpoints in terms of improving the efficiency of conference management, and there is room for further improvement in efficiency. In particular, by capitalizing the content discussed at the meeting and making it possible to share it across the organization, it is considered possible to further improve the efficiency of many meetings held within the organization.

Therefore, an object of the present invention is to provide a conference support system and a conference support program capable of capitalizing the contents of a conference and streamlining the operation of the conference.

The above and other objects and novel features of the invention will become apparent from the description and accompanying drawings herein.

A brief description of the representative inventions disclosed in the present application is as follows.

The conference support system according to a typical embodiment of the present invention is a conference support system that supports the operation of a conference in an organization, and is a video conference device that enables online video conference between a plurality of bases. An image recognition unit that recognizes an image related to each attendee by an image recognition technique from video data acquired by the video conferencing device, and audio data of each attendee acquired by the video conferencing apparatus are acquired, and the audio data is obtained. And, based on the comparison with the voice feature information of each attendee registered in advance and the motion information of each attendee acquired by the image recognition unit, the speaker of each remark in the voice data is specified. It has a voice recognition unit and a timeline management unit that outputs the voice data of each attendee acquired by the voice recognition unit as a timeline in the time series of remarks.

The present invention can also be applied to a conference support program that operates a computer as a conference support program as described above.

Among the inventions disclosed in the present application, the effects obtained by representative ones are briefly described as follows.

That is, according to a typical embodiment of the present invention, it is possible to capitalize the contents of the conference and improve the efficiency of the operation of the conference.

It is a figure which showed the outline about the structural example of each part of the conference support system which is one Embodiment of this invention. It is a figure which showed the outline about the configuration example at the time of operation of the conference support system which is one Embodiment of this invention. It is a figure which showed the outline about the screen example which is displayed on the display of the video conference apparatus in one Embodiment of this invention. It is a figure which showed the outline about the screen example which is displayed on the display of the video conference apparatus in one Embodiment of this invention. It is a figure which showed the outline about the screen example which is displayed on the display of the video conference apparatus in one Embodiment of this invention. It is a figure which showed the outline about the screen example which is displayed on the display of the video conference apparatus in one Embodiment of this invention. It is a figure which showed the outline about the screen example which is displayed on the display of the video conference apparatus in one Embodiment of this invention. It is a figure which showed the outline about the screen example which is displayed on the display of the video conference apparatus in one Embodiment of this invention. It is a figure which showed the outline about the screen example which is displayed on the display of the video conference apparatus in one Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, in all the drawings for explaining the embodiment, the same parts are in principle the same reference numerals, and the repeated description thereof will be omitted. On the other hand, the parts described with reference numerals in one figure may be referred to with the same reference numerals in the description of other figures, although they are not shown again.

<System configuration>
FIG. 2 is a diagram showing an outline of a configuration example of the conference support system according to the embodiment of the present invention at the time of operation. The conference support system 1 of the present embodiment is, for example, a server system 10 composed of a server device, a virtual server built on a cloud computing service, etc., and each base such as a conference room where attendees of the conference gather. In the example in the figure, it has a video conference device 20 installed at each of "base A" and "base B"). The server system 10 and each video conference device 20 are configured to be able to communicate with each other via a connection to a network 40 such as the Internet or an intranet.

The video conference device 20 is a device or system that enables online video conferences between a plurality of bases. For example, a large display, a camera, a microphone, various sensors, etc. such as the above-mentioned "Microsoft Surface Hub" can be used. A dedicated device provided can be used as appropriate. It may be configured by using a general-purpose PC (Personal Computer), peripheral devices such as a Web camera, a microphone, and a speaker connected to the PC, and existing software for video conference. Attendees can hold an online video conference while referring to images of attendees at other sites via the video conference device 20. At that time, by the function of the digital whiteboard, for example, the information handwritten by the attendees on the display configured by the touch panel can be shared between the bases and the attendees.

Attendees at each site can further connect to the conference support system 1 via the attendee terminal 30, which is an information processing terminal such as a PC (Personal Computer), a tablet terminal, or a smartphone. At the time of connection, for example, the video conference device 20 installed at its own base may be connected via a LAN (Local Area Network) or the like, or may be connected to the server system 10 via the network 40. good. Each attendee can display materials, input text information, and the like on the display of the video conference device 20 at each base by operating the attendee terminal 30, and share the information between the bases and the attendees. In addition, the information displayed on the display of the video conference device 20 can be referred to on the attendee terminal 30.

The server system 10 includes, for example, an OS (Operating System), a DBMS (DataBase Management System), a Web server program, etc. developed on memory from a recording device such as an HDD (Hard Disk Drive) by a CPU (Central Processing Unit) (not shown). By executing the middleware and the software running on it, various functions related to support for conference management including conferences between multiple bases will be realized.

FIG. 1 is a diagram showing an outline of a configuration example of each part of the conference support system according to the embodiment of the present invention. As described above, the conference support system 1 of the present embodiment has, for example, a configuration in which the server system 10 and the video conference equipment 20 installed at each base are connected to each other via the network 40.

The conference support system 1 includes, for example, a video conference processing unit 11 implemented by software, a voice recognition unit 12, an image recognition unit 13, an authentication unit 14, a timeline management unit 15, an evaluation / analysis unit 16, and a conference control unit 17. It also has each part such as a summary generation part 18. It also has data such as a user DB 101, a content DB 102, a timeline DB 103, and a minutes DB 104 implemented by a database (DB), a file, or the like. It should be noted that all or part of each of these parts and each DB may be centrally mounted on the server system 10, or may be individually provided by the video conference device 20, which is a system requirement. It can be flexibly configured accordingly (in the example of FIG. 1, it is not specified whether each part is mounted on the server system 10 or the video conference device 20).

The video conference processing unit 11 has a function of performing basic and general processing related to an online video conference between a plurality of bases. For example, the video and audio data of the base (meeting room) acquired by the camera and microphone included in the video conference device 20 are recorded in a file or the like and output to the display of the video conference device 20 of another base. Further, by the function of the digital whiteboard, materials, input information, and the like are displayed on the display of the video conference device 20 of each base and shared between the bases. These functions can be implemented by appropriately using existing software and systems that enable video conferencing, such as the above-mentioned "Skype for Business".

The voice recognition unit 12 records the voice data during the conference acquired and recorded by the video conference processing unit 11 in the content DB 102, converts it into text in real time by a known voice recognition technology, and identifies the speaker if possible. Then, it has a function of recording in the content DB 102 in association with the conference and the speaker (user). The method for specifying the speaker is not particularly limited, but for example, it is performed by comparing the feature information such as the voiceprint registered in advance in the user DB 101 (or acquired from the voice data during the meeting) with the voice data for each user. .. The direction in which the voice is emitted is specified based on the voice data acquired from a plurality of microphones, and the attendee information (and the (mouth) movement) in the video data specified by the image recognition unit 13 described later is used. Information) may be collated, and the audio data and the video data may be used in combination to specify the data.

The image recognition unit 13 records the video data (which does not have to include audio) of the base (meeting room) during the conference acquired and recorded by the video conference processing unit 11 in the content DB 102, and also a known image. It has a function to identify and grasp each attendee and their actions in real time by using recognition technology. If the conference support system 1 has acquired the information of the prospective attendees in advance, it may be specified from the prospective attendees. The identified attendee information (particularly face information) can be used, for example, in the attendee authentication process in the authentication unit 14 described later. Further, the information on the movements of the attendees is grasped as a gesture by the conference control unit 17 described later, and can be used for acquiring various information from the attendees during the conference and for controlling the operation of the conference by the attendees. can. In addition, the evaluation / analysis unit 16 described later can grasp information on the state of attendees such as dozing and posture.

In addition, in the content DB 102, in addition to the voice data acquired by the voice recognition unit 12 and the video data acquired by the image recognition unit 13, conference materials and whiteboard data acquired by the video conference processing unit 11 are stored in the content DB 102. , Recorded in association with each meeting. Specifically, for example, it holds a list in which the data of each content is associated with the target conference. Alternatively, it may be realized by associating using the file name related to the data of each content (for example, including the information that can identify the conference in the file name).

The authentication unit 14 has a function of performing authentication processing for the attendees of the conference. In the present embodiment, for example, an image of the face of the attendee specified by the image recognition unit 13 is previously displayed so that the attendee (user) does not need to be aware of the operation or process of authentication, which is unrelated to the conference itself. Face authentication is automatically performed by collating with the face image of each attendee registered in the user DB 101, and each attendee is specified. Not limited to the face, voice or the like may be collated. Of course, it is also possible to use a normal method of inputting a user ID / password on the attendee terminal 30, and for example, an ID card reading device or a biometric authentication device provided at the entrance of a conference room or the like. In cooperation with an external authentication system by the user, information indicating that the user has been authenticated may be acquired. For example, the conference control unit 17, which will be described later, may create a seating plan for the conference based on the information of the attendees identified through the authentication.

The timeline management unit 15 records the flow of remarks and comment postings by each attendee in the meeting in the timeline DB 103 and manages them in chronological order, and as shown in the screen example described later, the video conference device 20 It has a function to display in real time on the display.

The evaluation / analysis unit 16 has a function of performing various evaluations and scoring of the contents of the meeting in real time or after the fact. In addition to the entire conference, the evaluation targets include, for example, the individual remarks of each attendee included in the conference, and the attitude / state of the conference. In addition to the evaluation, for example, topics and frequently-used keywords can be analyzed from the contents of remarks during the meeting, and the voices, actions, postures, etc. of each attendee acquired by the voice recognition unit 12 and the image recognition unit 13 can be analyzed. It is also possible to analyze the emotions and conditions of the attendees from the information in. In addition, by analyzing the contents of the minutes accumulated in the minutes DB 104, it is possible to grasp what kind of content is being discussed at which meeting (meeting body), and the knowledge that can be used across the organization. It can also be used as a target asset.

The conference control unit 17 has a function of controlling the operation and progress of the conference. For example, screen control according to the format and mode of the meeting as described later, time management including the end time of the meeting, designation / suggestion of the next speaker according to the current topic and the number of comments of each attendee, etc. It may include control of breaks, off-records, ice breaks, etc. (that is, parts not left in the minutes) based on instructions / requests from attendees, and their cancellation. It may have a function of attaching a sticky note, a bookmark, or the like on the screen of the timeline, minutes, or other video conference device 20. Access control that limits the items and scope that may be disclosed about remarks and materials in the meeting based on the contents specified in advance for the entire meeting or the contents registered in the user DB 101 in advance for each attendee. May be done.

When the data related to the contents of the meeting recorded in the content DB 102, the timeline DB 103, and the minutes DB 104 are recorded on the video conference device 20, these data are stored in the server system 10 at the end of the meeting. You may control uploading to (or another management server, etc. not shown).

The summary generation unit 18 has a function of generating minutes by a known natural language process based on the text data of the contents of the meeting and recording the minutes in the minutes DB 104. Minutes may be generated collectively after the meeting is over, or may be generated in real time or at regular intervals so that each attendee can check them during the meeting. When ToDo or a task is set in the minutes, the contents may be output to an external task management system or workflow system (not shown) and linked. The contents stored in the minutes DB 104 can be appropriately referred to from the video conference device 20 and the attendee terminal 30.

<Screen example and function contents>
3 to 9 are diagrams showing an outline of a screen example displayed on the display of the video conference device 20 according to the present embodiment.

[At the start of the meeting]
The screen example of FIG. 3 shows a state in which attendees are recognized / authenticated and registered as attendees at the start of the meeting. In the area on the left side of the screen, the camera provided in the video conference device 20 displays an image of the conference room of the base where the video conference device 20 is installed, and all the attendees turn their faces to the camera. The image recognition unit 13 recognizes the face, and the authentication unit 14 performs face recognition and identification of attendees by collating with the face image data registered in advance in the user DB 101. If the conference support system 1 has acquired the information of the prospective attendees in advance, the prospective attendees may be identified by face recognition.

As shown in the figure, face recognition may be performed collectively with a plurality of attendees gathered, or may be performed individually when each attendee enters the conference room. good. In addition to face recognition, as shown in the figure, the voice data registered in the user DB 101 in advance may be collated for authentication.

The information of the attendees whose faces have been recognized is displayed as a list in the area on the right side of the screen, for example, by an image such as an icon indicating each attendee or a face mark. In the example of FIG. 3, among the attendees assigned the numbers "1" to "4" in the area on the left side of the screen as a result of face recognition, the attendees assigned "4" in the area on the right side of the screen. Indicates that face recognition could not be performed and could not be identified. In this case, you can manually register as an attendee by selecting the "Register new" button.

Manual registration may be performed, for example, by reading an ID card such as an employee ID card or an admission card with a reader device or the like. Alternatively, the attendee may register by logging in to the conference support system 1 via an application installed on the attendee terminal 30 such as a smartphone owned by the attendee. You may register by directly searching the user DB 101 using your name, employee number, or the like as a key.

When manually registering attendees, for example, if the attendee does not have an ID card or the like and is not registered in the user DB 101, another attendee who has already been authenticated or is granted authority in advance. It may be possible to manually register by the user who has been authenticated. For example, in the example of FIG. 3, an attendee assigned a number of "1" who has already been authenticated and registered as a facilitator can still select the "Confirm unauthenticated attendees" button (not shown). Unauthenticated attendees (attendees assigned the number "4" in the example of FIG. 3) may be displayed in a list and manually registered individually or collectively.

Attendees from outside (other bases) are also authenticated by logging in to the conference support system 1 and registered as attendees. Images such as icons and face marks indicating attendees who have been authenticated and registered are displayed on the attendee terminal 30 of each attendee in addition to the screen of the video conference device 20.

At the time of attendee authentication and registration, the characteristic data of each attendee is acquired by the voice recognition unit 12 and the image recognition unit 13 until the start of the conference so that each attendee can be identified from the video and audio during the conference. It may be associated with the identified attendees and remembered. For example, in addition to the faces of each attendee on the day of the meeting, information such as voice, features such as clothing color, hair color and length, and the presence or absence of glasses is stored as feature data. Even after the start of the meeting, these feature data may be appropriately acquired and updated.

At the start of the meeting, representatives such as the facilitator (“Ichiro Suzuki” to which “1” is assigned in the example of Fig. 3) and the moderator will share and disclose the contents of the meeting (statements, materials, etc.). You may set it. At that time, specific contents may be set individually, but the types of meetings (for example, "idea generation", "subcommittee", "business evaluation meeting", "business promotion committee", "head office general meeting"" Etc.), and the disclosure range (for example, "company-wide", "attendee-only", "affiliation-only", "only those with predetermined authority", etc.) should be patterned in advance and selected from them. You may try to reduce the setting load with. The user may manually and dynamically modify the pattern, or a new pattern may be created on the spot. It may be set at the time of convening the meeting instead of at the start of the meeting.

[During (and after)]
In the screen example of FIG. 4, as an example of the screen during the meeting, an example of a timeline for recording and managing the remarks of each attendee is shown. Images such as icons and face marks indicating the attendees of the meeting are displayed side by side at the top of the screen, and the text of each statement is displayed as a timeline in the vertical column corresponding to each attendee. It is displayed in series. In addition, statistical information such as the number of remarks and the remark time of each attendee is also displayed. Attendees who have a large number of remarks or a long remark time may be highlighted by enlarging or darkening the image such as an icon or a face mark indicating the attendee.

The display method of the timeline during the meeting is not limited to the display in which each attendee is arranged in a horizontal row as shown in FIG. 4, for example, the icon of each attendee is actually displayed in the meeting room. It may be displayed at a position corresponding to the seat arrangement of. At that time, the actual seating arrangement may be displayed as it is (for example, if the seat is seated on a circular table, the seating arrangement may be arranged in a circle), or the seating order and position may be displayed on the edge of the screen. The icons and the like may be arranged side by side. In either case, the attendees, especially those at other locations, can intuitively grasp the seating position of the attendees. These display methods may be switched according to the instructions of the attendees.

When each attendee moves a seat, the display position of the corresponding icon or the like may be automatically changed. Further, the display of the icon or the like of each attendee may be individually hidden based on the instruction from the attendee. If there is a misrecognition that the actual seating position of the attendee does not correspond to the display of the icon or the like on the screen, for example, the target icon or the like is moved by an operation such as dragging to the correct position. At that time, authentication processing such as face recognition and registration as an attendee may be performed again.

The identification of the speaker of each utterance during the meeting is, for example, with characteristic information such as a voiceprint registered in advance in the user DB 101 by the voice recognition unit 12 or extracted from the voice data of each attendee acquired at the start of the meeting. , It is performed by comparing with the feature information of the voice data related to the target utterance. In addition to this, it may be performed based on the information on the state of each attendee specified by the image recognition unit 13 from the video data during the meeting.

For example, the movement of the mouth and body of the attendee specified by the image recognition unit 13 may be analyzed from the video to identify the speaker. For example, an attendee whose mouth is moving when speaking or whose body is making movements peculiar to speaking (gestures, gestures, vertical movement of shoulders, etc.) is specified as a speaker. If a plurality of attendees including the actual speaker do not face the video conference device 20 (the camera provided therein) and cannot acquire a facial image, for example, first, the mouth of the attendee who can acquire the facial image. Determines if is moving, and if not, excludes these attendees from the candidates for speakers. Then, among the remaining attendees, the attendees whose bodies do not move peculiarly at the time of speaking are further excluded, and the remaining attendees are presented on the screen as candidates for speaking. For example, a list of speaker candidates may be displayed on a pop-up screen, or icons and the like related to speaker candidates may be highlighted.

Not limited to the speaker, for example, for the attendee who wrote on the display (touch panel) of the video conference device 20 by the function of the digital whiteboard, the attendee is identified from the video data of the attendee who is writing. You may.

When the speaker can be identified, for example, a balloon or a mark of "speaking" is added to the image such as an icon related to the specified speaker in the timeline shown in FIG. 4, or an icon or the like. You may change the display form of to indicate that the target attendee is speaking. If the attendee who is indicated to be speaking is not actually the speaker and another attendee is the speaker (in the case of misrecognition), for example, an icon related to the true speaker is displayed on the screen. You may be able to correct the misrecognition of the system by selecting it by tapping it on the top and selecting the "speaking" item from the pop-up menu.

In the voice recognition unit 12, in addition to specifying the speaker based on the voice data, the content of the speech is converted into text in real time by the voice recognition technology and recorded in association with the specified speaker. As a result, as shown in FIG. 4, the timeline management unit 15 can display text information in real time as the remarks of the target attendees on the timeline. It is also possible to automatically generate minutes based on this textual information. If the automatically associated speaker is misrecognized, for example, select another candidate from the list of candidates on the pop-up screen that is displayed by selecting the target text on the screen and select the speaker. Can be corrected.

The content of the remark is not limited to the textualized voice data during the meeting, and may include the textualized content based on other contents and information. For example, it may be converted into text based on actions (sign language, gestures, etc.) based on a specific law of attendees. For example, when a user uses sign language, information such as a flag to "convert an action (sign language) into text (instead of voice)" is set in advance as information related to the user in the user DB 101. During the meeting, the process of converting the operation (sign language) recognized by the image recognition unit 13 into text is automatically performed based on this setting information.

Alternatively, for each user, "information used as a source for text conversion" may be set in advance in the user DB 101 by a code value or the like. For example, various speaking methods such as "voice only (default)", "voice + motion (sign language)", "motion (sign language)", and "eye movement" can be set for each user. As a result, even when various attendees hold one meeting at the same time, they can attend without feeling inconvenience.

When a call, message, email, etc. arrives at the attendees during the meeting, for example, a balloon or mark "Notice" is added to the image such as the icon related to the target attendee, or the icon etc. The display form may be changed to inform the attendees. Further, the sensor provided in the video conference device 20 that another user different from the current attendee has entered the conference room during the conference. When detected by, the image recognition unit 13 analyzes whether or not the user has face-authenticated before the conference, and if the user has been authenticated (that is, when the user is re-entering the conference room), nothing is done. .. On the other hand, in the case of an unauthenticated user, a video conference is displayed on the screen for authentication and registration using, for example, face authentication or authentication by an ID card such as an employee ID card or an attendee terminal 30 such as a smartphone. It is displayed on the device 20 and the user is authenticated by the method selected by the user. If possible, the authentication process may be automatically performed by either method. Depending on the nature and purpose of the conference, authentication for users who attend late before the start of the conference is performed using other terminals such as the attendee terminal 30 that cooperates with each other, not through the video conference device 20. It may be possible to set it so that it can be done.

In this embodiment, it is possible to support not only a proceeding-type meeting but also various types of meetings. The screen example of FIG. 5 shows an example of a function of assisting attendees in a brainstorming type conference. For example, the area on the left side of the screen shows an example of a digital whiteboard that allows information to be shared among attendees. On this screen, in addition to operations such as inputting text information (which may be by voice recognition) and displaying materials, each attendee can freely attach and remove sticky notes for brainstorming. It can be performed. In addition, for words and phrases recognized as keywords on the whiteboard, in the area on the right side of the screen, information related to the keywords is automatically acquired from various information sources in real time and displayed by attendees. Reduce the burden of collecting this information on your own and help you focus on the discussion.

As the information source, for example, in addition to various websites on the Internet, search engines, SNS services, and the like, information accumulated in the company may be used. At that time, the past meeting contents recorded and accumulated in the minutes DB 104 or the like of the meeting support system 1 can be used as an asset. For example, the meeting control unit 17 or the like extracts information such as minutes and files related to keywords from the minutes DB 104 and the like, presents them as shown in the figure, and makes them easily accessible. .. This makes it possible to obtain information about what is currently or in the past being discussed internally, and to avoid duplicate meetings and discussions. In addition, it is possible to easily grasp and contact people and organizations that possess necessary information and skills.

When searching the minutes DB 104 or the like, for example, all the minutes stored in the minutes DB 104 may be searched by using keywords in the current meeting or the meeting scheduled to be held in the future. The search may be limited to the minutes of meetings that are in progress or related to upcoming meetings. This makes it possible to reduce search noise, improve accuracy, and perform an efficient search. For example, the code value that identifies the meeting type / type for each meeting (for example, "01" for "○○ system regular meeting", "02" for "△△ system regular meeting", etc.) is automatically or manually performed. If so, this code value can be used to identify the relevant conference. Not limited to the code value, it is also possible to specify the related conference from other information such as the conference name and the breakdown of attendees.

Further, the information on the contents of the past meeting recorded / accumulated in the minutes DB 104 or the like may be used to determine whether or not a new meeting is to be held after that and the implementation format thereof. For example, through a data input / output interface (not shown), information such as the agenda of the meeting newly considered to be held, the list of prospective attendees, and the materials to be referred to is acquired from the user and recorded in the minutes DB 104 or the like. In combination with the accumulated data of the contents of past meetings, the necessity and format of holding the target meeting (for example, broadcast format, circulation / rotation format by e-mail, posting on the website on the intranet) The format of only) may be automatically determined by the evaluation / analysis unit 16 or the conference control unit 17 and presented to the user.

In addition, based on the remarks of each attendee in the past meeting contents recorded and accumulated in the minutes DB 104, etc., and the attribute information registered in the user DB 101, attendance is "required" for each prospective attendee. You may decide "arbitrary" or "unnecessary". In addition, the time required for a new meeting to be held is estimated, and the schedule and schedule of each scheduled attendee in other schedule management systems and schedule management systems are collated to determine an appropriate date and time for the user. It may be presented. Further, when the user approves the presented date and time, the schedule of the target conference may be automatically registered as the schedule or schedule of each prospective attendee. The determination process as described above may be performed based on, for example, predetermined rules and conditions set in advance, or may be performed by using known techniques such as machine learning and artificial intelligence.

During the meeting, the evaluation / analysis department 16 further aggregates and analyzes trend information such as what topics and keywords are currently interested in and discussed within the company, as shown in the screen example in FIG. It may be done in real time and the results may be displayed on the video conference device 20 for sharing among attendees. For example, keywords that frequently appear in the contents of intranet sites, keywords that are frequently specified by each employee when using a search engine, keywords that frequently appear in various internal meetings (minutes), etc. are aggregated. Analyze and display. As a result, it is possible to obtain a wider range of information about the contents currently or in the past being discussed in the company, and it is possible to avoid duplicate meetings and discussions. It should be noted that such support for meetings by capitalizing in-house meeting information can be similarly provided not only for brainstorming type meetings but also for other types of meetings.

The screen example of FIG. 7 shows an example of a function of supporting attendees in a broadcast-type meeting, for example, in which the department manager explains the department status to each member in the department all at once. For example, in the area on the left side of the screen, the camera image of the speaker / explainer such as the director is displayed in the upper area, and the lower area is used for explanation or materials such as figures and tables for supplementing the explanation. Is displayed. Further, in the area in the center of the screen, a list of attendees of the conference, that is, attendees listening to the remarks and explanations of the speakers displayed on the camera image and their whereabouts is displayed.

Further, in the area on the right side of the screen, the process of speech by each attendee or by text input via the attendee terminal 30 is displayed as a timeline. In the example of FIG. 7, since the number of attendees is large, the remarks (comments) of each attendee are put on the timeline of one column instead of the timeline in which the columns are independent for each attendee as shown in FIG. It is in the form of being mixed in chronological order.

[Making minutes]
As described above, the voice data related to the remarks of each attendee during the meeting is converted into text in real time and recorded in the content DB 102. On the other hand, the summary generation unit 18 creates minutes in real time (or after the meeting) in the form of a summary (which may include figures and tables) using existing document summarization technology and the like, and targets the minutes. It may be recorded in the minutes DB 104 in association with the information for identifying the meeting (for example, the meeting name, the date, etc.).

The screen example of FIG. 8 shows a screen for referring to the created minutes, and the bibliographic items of the minutes and the automatically summarized agenda contents are displayed in the area on the left side of the screen. In addition, a list of tasks decided and created at the meeting, information on deadlines and assigned persons in charge are displayed in the upper part of the area on the right side of the screen, and the referenceable range of the minutes is displayed in the lower part. The contents set automatically or manually are displayed.

In the display of the summarized agenda, it is possible to display not only the text information but also the form having various additional functions. For example, the evaluation / analysis unit 16 identifies important keywords based on the strength of the speaker's remarks, the content of modifiers, the content of past meeting records, etc., and the character color of the corresponding part in the text of the minutes. , Font, typeface, etc. may be changed to highlight.

Further, for example, even if the voice recognition unit 12 or another data conversion unit (not shown) has a function of converting the text of the minutes into voice data and playing it back at any time during or after the meeting. good. For example, the minutes can be reproduced by selecting an icon displayed on the screen of the video conference device 20 or the attendee terminal 30 of the attendees. In addition, tags, bookmarks, and the like may be attached to the contents of the minutes so that only the part corresponding to the tag specified by the user can be reproduced at the time of audio reproduction.

The text of the minutes is not limited to conversion to voice data, and may be converted to another format by a data conversion unit or the like (not shown). For example, by converting to Braille data, a visually impaired user may be able to check the contents of the minutes. Further, it may be translated into one or more other types of languages, and the translated content may be provided to the user by text, voice data, or a combination thereof. The language of the translation destination may be arbitrarily specified by the user, for example, depending on the attributes of the attendees of the target meeting and the user who has the authority to view the minutes (for example, acquired from the user DB 101 or the like). The optimum language may be automatically determined.

During the meeting, the meeting control unit 17 automatically or manually identifies unnecessary parts (for example, ice breaks, inappropriate remarks, NG words, etc.) in the meeting contents, and leaves these contents in the timeline and minutes. You may not have it. For example, an "ice break" icon or button (not shown) is arranged on the screen of the timeline as shown in FIG. 4, and while the attendees select and enable it (for example, touch the icon or the like). It may be enabled only while the operation is continued, or it may be enabled / disabled for each touch.) Conversations and remarks are controlled so as not to be recorded in the timeline or minutes. Even in this case, the original audio or video data may be recorded in the content DB 102, another system / database for the administrator, or the like.

Instead of the valid / invalid instruction by an explicit method such as the operation of an icon or the like on the screen, another less conspicuous instruction method may be used. For example, if an attendee turns the strap of the employee ID card around his neck behind his neck, it will stop recording on the timeline or minutes as an off-record, and if it is returned to the front, it will resume. You may give a valid / invalid instruction.

Further, the conference control unit 17 may automatically convert a specific word or content among the contents such as remarks during the conference into an expression such as an abbreviation or black paint. For example, when the master information of the abbreviation of the customer company name (for example, a character string consisting of two or three initial letters of the alphabet) is stored in advance and a specific customer company name appears in the remarks during the meeting, This may be automatically converted into an abbreviation and recorded on the timeline or minutes. In addition, if it corresponds to a pre-registered NG word, it may be masked by converting it to black on the timeline or minutes.

These conversions and masks may be performed in real time during the conference or separately after the conference. When the meeting is held after the meeting, for example, the place where the amount of money is described or the place where a specific name is described may be uniformly converted to black and masked according to the characteristics of the target meeting. However, it may be possible to convert only the expressions for which the user has the authority to convert.

In addition, if there is a term that the attendees do not understand or want to know the related field in the timeline or minutes, for example, by selecting the term by touch etc., the meaning of the term and related terms are related. Information on articles (titles, summaries, links, etc.) may be displayed on the attendee terminal 30 of the attendee. These information can be obtained from, for example, various websites, search engines, and the like. If the attendee wants to share this information with other attendees, he / she may display and share the information on the attendee terminal 30 or the video conference device 20 of the other attendees according to the instruction. Links to this information may be recorded in association with the target points in the timeline or minutes.

During or after the meeting, each attendee can attach bookmarks, tags, emotional flags, sticky notes, simple memos, etc. to the minutes. For example, on the screen of the timeline as shown in FIG. 4, the attendee performs a predetermined method or gesture such as touch or long press on the image of the icon or face mark indicating each attendee, or By giving an instruction via the pop-up menu, the timeline management unit 15 can easily bookmark the current content of the statement on the timeline, record one's own emotions, and the like. This information is taken in when the minutes are created and reflected in the minutes.

Further, a bookmark, a flag indicating emotions, a sticky note, or the like can be directly attached to the text in the minutes via the reference screen of the minutes as shown in FIG. The evaluation / analysis unit 16 may detect a part with a sticky note or a flag of good feeling / bad feeling by many attendees, and may highlight the part as an “important point”. The minutes may be further automatically summarized by the summary generation unit 18 centering on the points recognized as “important points” and shared by all the attendees of the meeting.

A bookmark, a flag indicating an emotion, or the like may be attached by detecting a predetermined action or gesture of an attendee by the image recognition unit 13. For example, the correspondence between the contents of bookmarks and emotion flags and actions and gestures is patterned and stored in advance. For example, by associating the "thumbs up" gesture of the user's hand with the emotion flag "like", the video conference device 20 indicates that the attendee took the "thumbs up" gesture during the conference. When detected by the camera or sensor of the above, information such as an image related to the emotion flag of "Like!" Is displayed at the place related to the target remark on the timeline. The emotion flag may be associated with the target part of the minutes and recorded in the minutes DB 104.

It is not limited to movements and gestures having a general meaning such as "thumbs up", but may be expressed by meaningless movements and combinations thereof. For example, a bookmark may be added by repeating the operation of "gooper" that ties the palms together and opens it twice. Further, the content of the voice instruction of the attendee may be recognized by the voice recognition unit 12 and a corresponding bookmark, emotion flag, or the like may be attached, not limited to the movement or the gesture.

Further, for example, the video data of the base (meeting room) acquired by the camera or microphone included in the video conferencing device 20 is recorded as a still image at a time specified by the user by a predetermined method, and " It may be managed by adding a "photo" flag. Further, for example, the screen displayed on the display of the video conference device 20 may be recorded as a still image by a predetermined method, and may be managed by adding a “capture” flag to the screen. For image data managed by the "Photo" and "Capture" flags, only the data selected by the user during or after the meeting by a predetermined method (selection with a check box, touch, long press, etc.) May be reflected in the minutes.

Insertion information such as bookmarks and tags, emotion flags, sticky notes and simple memos given / reflected in the minutes may be referenced and edited only by the person who gave them, or other attendees or non-attendants. The user may also be able to refer to and edit. These authority settings may be set by the user who has given the target insertion information. On the contrary, the user who refers to these insertion information may also be able to display only the insertion information given by the user specified by himself / herself. With these, it is possible to filter unnecessary insertion information so that it cannot be seen.

As shown in the example of FIG. 8, tasks may be assigned to the contents of the minutes in addition to bookmarks and sticky notes. The evaluation / analysis unit 16 analyzes the contents before and after the agenda (text, audio, video, materials, etc.) for which the task is set, and the evaluation / analysis unit 16 automatically sets the person in charge to be assigned, and other tasks. The deadline and task content may be automatically registered in the management system or workflow system as the schedule of the person in charge. If the person in charge cannot be set automatically, the user may identify and set an appropriate person in charge by searching as shown in the figure.

Further, it may be determined whether the task is group work or individual work and given as attribute information. At this time, in the case of individual work, the candidate for the person in charge may be recommended based on the attributes of the attendees and the content of the remarks during the meeting, and in the case of group work, the candidate for the main person in charge may be recommended. In addition, the main person may be recommended to list the candidates who can be members of the group.

The minutes displayed on the screen of the video conference device 20 during the meeting may be displayed on the attendee terminal 30 by each attendee and may be shared. Then, if the content of the minutes displayed on the attendee terminal 30 is incorrect or inappropriate, the attendee may correct and overwrite the minutes through the operation on the attendee terminal 30. can. Only the speaker about the content of the subject and the user who has been granted the authority in advance (for example, a member of the secretariat) may be able to modify it.

The method of correction is, for example, when the user selects the target part in the minutes, a pop-up menu (not shown) is displayed, and when "correction" is selected from the executable processes in the menu, the target part is corrected. Make sure that a pop-up window is displayed for you to correct and enter the text of. When the user corrects the text and presses the "OK" button, the text of the target part in the minutes is replaced with the corrected content.

In a brainstorming type meeting as shown in the example of FIG. 5 above, the attendees may be allowed to freely modify or add comments to the minutes. The added corrections and comments are displayed on a screen separate from the original minutes as a provisional one, and when the attendees perform an operation or instruction to the effect of confirmation, the original minutes are displayed. On the other hand, it may be reflected by overwriting.

As described above, in the minutes DB 104, for example, the contents of the past minutes including those of other conference bodies in the company are widely recorded and accumulated as a history. By designating and searching for the minutes DB 104 during or after a meeting by designating an arbitrary keyword, the minutes of another meeting in which the target keyword frequently appears may be extracted and displayed. The minutes that can be displayed are limited to those that are permitted to be published, and the availability of publication may be controlled by the user's authority. If the extracted minutes are from another meeting currently being held, information such as the venue, attendees, contact means, etc. may be acquired and displayed. This allows, for example, to avoid unnecessary meetings when other meetings with duplicate or similar contents are being held, or conversely, to hear reference information and share information. , It is possible to improve the efficiency of conference management.

After the meeting, in addition to the attendees, authorized non-attendants may be able to insert audio or text comments on the timeline and minutes of the finished meeting. For example, you may add a comment that supplements the content of the remarks during the meeting, or the supervisor reviews the content (output) of the minutes of the meeting that requires output, such as idea generation and planning business meeting. , The review content may be recorded in association with the minutes. In addition, the workflow of the approval process for the reviewed minutes may be automatically generated.

As described above, the minutes recorded in the minutes DB014 can be shared within the company or between organizations, but depending on the sharing method, for example, the contents to be reflected in the minutes by the user's setting. It may be possible to suppress the number of minutes or automatically change the structure of the minutes. For example, when the reference screen of the minutes as shown in FIG. 8 is displayed on the screen of the video conference device 20 or on each attendee terminal 30, it is managed by the above-mentioned "photo" and "capture" flags. The display of the image data may be suppressed, or the image data may be processed and processed by a method set in advance by the user before being displayed.

Further, the image data managed by the "photo" and "capture" flags is recorded with respect to the minutes data output as data when the user presses the "download" button in the screen example of FIG. It may be inserted in the appropriate place of the minutes according to the time when it was done. Alternatively, among a plurality of minutes templates registered in advance by the user via a data input / output interface (not shown), the minutes template specified by the user or the minutes template automatically recommended by the conference support system 1 is specified. You may insert it in the place of.

Further, when the user presses the "send mail" button in the screen example of FIG. 8 to send the minutes data to another user by e-mail and share it, the meeting name, the date and time of the meeting, and the meeting are sent to the e-mail. The minimum content such as the summary of the above may be displayed. In addition, instead of transmitting the minutes data itself, a URL (Uniform Resource Locator) for accessing the conference support system 1 is displayed by a hyperlink, and another user who receives the e-mail displays the above URL. You may be able to view the minutes of interest by selecting the link.

[Efficiency of meeting management]
As another means for improving the efficiency of the operation of the meeting, for example, the meeting control unit 17 identifies the attendees who should speak in consideration of the contents of the meeting in progress and the attribute information such as the background of each attendee. The attendees may be displayed to encourage them to speak, or a list of candidates for attendees to speak may be displayed on the attendee terminal 30 of the facilitator or the moderator. When displaying the list of candidates, the information on the achievements with / attached the emotion flag in the past, the evaluation value of the degree of contribution in the past meeting, and the like may be additionally displayed as reference information.

Attendees may wear so-called smart glasses in place of or in addition to the attendee terminal 30 at the time of the meeting. As a result, for example, the content displayed on the video conference device 20 or the attendee terminal 30 is linked to the smart glasses, or the content (image) seen by the attendees by the smart glasses is linked to the video conference device 20. Can be done. In addition, for example, information on each attendee (for example, public attributes and careers, names of meetings attended in the past, etc.) is displayed on smart glasses (or attendee terminal 30) before or during the meeting. It may be possible to refer to and confirm.

For the image of each attendee's icon or face mark, for example, the conference control unit 17 or the like may display an emotion icon indicating the emotion or state of the attendee in the vicinity thereof. The image of the attendee's icon or face mark itself may be changed according to the emotion or state. For example, if the meeting is prolonged and the end time is expected to exceed the attendee's scheduled leaving time on the day or the maximum overtime hours of the current month, an emotional icon with an impatient expression may be displayed or the attendee's emotion icon may be displayed. Change the color of the icon from green to red. Instead of automatically changing the display content, each attendee may be able to change the display content by an operation from the attendee terminal 30 or the like.

The content displayed by the emotion icon may include information indicating an impression about the room temperature of the conference room, such as "hot", "cold", "just right", and "windy". The content of the emotion icon selected by each attendee during the meeting may be aggregated in real time by the conference control unit 17, and the air conditioning setting of the conference room may be automatically changed based on the aggregation result. Not only the impression about the room temperature of the conference room, but also the impression about other environment where the setting can be automatically changed / adjusted such as the brightness of the lighting of the conference room.

[Meeting evaluation / analysis]
During or after the meeting, the content of the meeting may be evaluated and analyzed by the evaluation / analysis unit 16 and the result may be displayed on the video conference device 20 or the attendee terminal 30. For example, a summary or important keywords may be extracted from the timeline or minutes, and the obtained text may be subjected to positive / negative analysis to calculate an evaluation score and displayed.

In addition to the content of statements in the timeline and minutes, for example, the facial expressions and movements of the attendees grasped by the image recognition unit 13 (sleeping, doing work not related to the meeting, etc.), attendees are attached. The situation of the meeting may be analyzed based on the emotional flag, the message, etc., the evaluation score may be calculated for each item, and this may be displayed. For example, as shown in the screen example of FIG. 9, the evaluation score of the entire conference is displayed in the center, and the evaluation score of each item is displayed in the periphery. The evaluation contents in real time may be displayed as a dashboard on the screen during the meeting as shown in the example of FIG. 4 above.

In addition, for example, by specifying the title, tag, keyword, period, etc. of the meeting, the relevant past and ongoing meetings are extracted, summarized, and sorted in chronological order to change the remarks and opinions. Information such as the transition, the transition of the score of the positive / negative analysis, the conclusion of the meeting (decision, etc.) may be displayed. For meetings held multiple times by the same conference body, such as regular meetings, information such as a summary of each meeting and distribution of emotion flags is aggregated and analyzed to score efficiency, and the score changes for each meeting. May be displayed as a dashboard.

In addition to the evaluation of the meeting, the degree of contribution of the attendees to the meeting may be evaluated. For example, during a meeting, when the tempo of each attendee's speech changed from slow to high, or when each attendee gave a certain number of "tired" or "boring" emotion flags. It is detected that the state of the conference has changed from a low state to a good state, such as when the state has changed to a state in which a certain number or more of "like" emotion flags are attached. Then, a plurality of attendees who have spoken during the transition are identified, keywords included in the content of each identified attendee's speech are weighted based on predetermined conditions, and scored for each attendee. It can be the contribution of attendees. The keyword weighting may be automatically set, for example, in descending order of relevance (for example, frequency of appearance in the text) with the minutes of the entire meeting or the text of the summary.

Attendees' contributions to the meeting may be assessed based on the number of emotion flags assigned by other attendees to each attendee's remarks. In addition, identify attendees who are considered to have a low contribution to the meeting, such as being asleep, having no remarks, or performing work unrelated to the meeting. The evaluation may be made after considering attribute information such as whether or not there is, whether or not it is related to the contents of the meeting, and whether or not it is attending on behalf of the person.

In addition, for a certain attendee, even though the speech time is long (the number of speeches is large), there are many negative emotion flags, attendees doing other work and attendees who are dozing off, etc. If it is evaluated that the efficiency of the video conference device is reduced, a comment or a message urging another speaker or another speaker may be displayed on the screen of the video conference device 20. For example, specify other attendees such as the conference moderator and facilitator to output messages and voices such as "What do you think about Mr. XX?" And "Would you like to take a break?" You may. In addition, regardless of the evaluation and contribution to the remarks of a specific attendee, for example, when a specific emotion flag such as "tired" or "boring" is attached by a certain number of attendees, "temporary break". You may output a message such as "Would you like to do it?" Or voice.

As described above, according to the conference support system 1 which is an embodiment of the present invention, attendance is held in a conference held within an organization such as an in-house, particularly in an online conference by voice or video between a plurality of bases. It is possible to automatically perform processing such as authentication processing and identification of persons, creation of timelines and minutes, evaluation of meeting contents, etc., which can support and improve the efficiency of meeting management.

In addition, by capitalizing the content of meetings discussed by multiple conference bodies and making it possible to share them across the organization, it is possible to obtain information on the content currently or in the past being discussed internally, which is duplicated. It is possible to avoid meetings and discussions that have been held. In addition, it is possible to easily grasp and contact people and organizations that possess necessary information and skills. As a result, it is possible to further improve the efficiency of many meetings held within the organization.

Although the invention made by the present inventor has been specifically described above based on the embodiment, the present invention is not limited to the above embodiment and can be variously modified without departing from the gist thereof. Needless to say. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to add / delete / replace a part of the configuration of the above embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

The present invention can be used for a conference support system and a conference support program that support a conference between a plurality of bases.

1 ... Meeting support system,
10 ... Server system, 11 ... Video conference processing unit, 12 ... Voice recognition unit, 13 ... Image recognition unit, 14 ... Authentication unit, 15 ... Timeline management unit, 16 ... Evaluation / analysis unit, 17 ... Conference control unit, 18 ... Summary generator, 20 ... Video conference system, 30 ... Attendee terminal, 40 ... Network, 101 ... User DB, 102 ... Content DB, 103 ... Timeline DB, 104 ... Minutes DB

Claims (9)

A conference support system that supports the operation of conferences in an organization.
A video conference device that enables online video conferences between multiple locations, and
The voice data of each attendee is acquired by the video conferencing device, and the speaker of each remark in the voice data is specified based on the comparison between the voice data and the voice feature information of each attendee registered in advance. Voice recognition unit and
It has a timeline management unit that outputs the voice data of each attendee acquired by the voice recognition unit as a timeline in chronological order of speech.
The timeline management unit is a conference support system that emphasizes an image showing the attendees according to the number of remarks or the amount of remarks of the attendees in the output timeline. In the conference support system according to claim 1,
Further, a summary generation unit that summarizes the contents of the timeline by natural language processing, acquires minutes, associates them with information related to the target meeting, and records them in the minutes recording unit.
Relevant from the minutes recorded in the Minutes Recording Department, including other meetings held within or currently being held within the organization, based on the timeline or keywords extracted from the minutes. A conference support system that has a conference control unit that acquires and outputs information related to minutes. In the conference support system according to claim 2,
The timeline management unit is a conference support system that adds information related to the emotions of the attendees to the timeline and / or the minutes based on instructions from the attendees. In the conference support system according to claim 1 or 2.
In addition, it has an evaluation / analysis unit that evaluates and outputs the content of the meeting and / or the contribution of each attendee to the meeting.
The evaluation / analysis unit is a conference support system that evaluates and outputs the degree of contribution of each attendee to the conference based on the remarks of each attendee acquired by the voice recognition unit. In the conference support system according to claim 1,
An image recognition unit that recognizes the faces of each attendee by image recognition technology from the video data acquired by the video conference device, and
It has an authentication unit that performs an authentication process for identifying each attendee by collating the face recognized by the image recognition unit with a face image registered in advance.
The authentication unit is a conference support system that accepts manual registration of attendees different from the attendees specified by the authentication process. In the conference support system according to claim 1,
The timeline management unit is a conference support system that accepts manual corrections of speakers specified by the voice recognition unit. In the conference support system according to claim 1,
The timeline management unit is a conference support system that converts a predetermined specific word or content among the contents of remarks during a conference into another expression in the output timeline. In the conference support system according to claim 7,
The other expression is an abbreviation or mask, a conference support system. A conference support program that lets a computer perform processing so that it functions as a conference support system that supports the operation of a conference in an organization.
The audio data of each attendee is acquired by a video conferencing device that enables online video conferencing between multiple bases, and the audio data is compared with the audio feature information of each attendee registered in advance. Based on the voice recognition process that identifies the speaker of each statement in the voice data,
A timeline management process that outputs the voice data of each attendee acquired by the voice recognition process as a timeline in chronological order of speech, and
A conference support program for causing the computer to perform an enhancement process for emphasizing an image showing an attendee according to the number of remarks or the amount of remark time of the attendee in the output timeline.

Images