JP2018073237A - Conference information display system, conference information display method and conference information display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make content of a conference based on a record easily visibly displayed to a user when the conference is recorded and the content of the conference is displayed later.SOLUTION: A conference information display system comprises: a storage device which stores at least any of documents, images of materials, voice and images of conference participants as conference information objects; and a display device which displays at least any of the documents and images of the stored conference information objects, and is provided with: a storage processing part which associates each time zone obtained by dividing a period of the conference into a plurality of time zones with the conference information objects to be stored in the storage device; a calculation processing part which calculates degrees of excitation in each time zone of the conference based on the stored conference information objects; and a display processing part which displays the stored conference information objects on the display device at degrees of integration according to the degrees of excitation in each time zone, and in time-series arrangement.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a conference information display system, a conference information display method, and a conference information display program. More specifically, the present invention relates to a conference information display system, a conference information display method, and conference information capable of storing the content of a conference and displaying the content of the conference at a later date. It relates to the display program.

  As a conference system, a system for electronically recording the contents of a conference is known. However, in general, it only saves images and audio during the meeting. There is a method of referring to the minutes manually created by the recorder when looking back on the contents of the meeting later. However, it takes time to create the minutes. Further, when trying to look back on the contents that are not recorded in the minutes, it is necessary to review all the recordings during the meeting (for example, voice recordings and videos of meeting participants), which takes a long time.

  On the other hand, when a conference participant uses his / her mobile phone to access the conference system and finds a statement that is important during the conference, or there is a valid statement to be confirmed later. For example, a system is proposed that extracts whether an utterance is an important utterance or an utterance that attracts attention by instructing insertion of an electronic bookmark when the conference is in progress (see, for example, Patent Document 1). .

  Furthermore, the person's biological information is synchronized with the person's video information, and a state estimation value indicating the person's mental state is obtained using a predetermined evaluation function based on the biological information or voice information at the time of photographing. An information processing apparatus has been proposed that can look back on the content including the emotions of conference participants, etc. by arranging and displaying objects representing the position in a coordinate space expressed using the mental state of the person (For example, refer to Patent Document 2).

JP 2005-109929 A JP 2006-6268395 A

However, although the thing of patent document 1 can specify the timing when there was an important statement etc., in order to understand the content of a meeting, it is necessary to review the timing and the agenda content before and behind that.
According to the technique disclosed in Patent Document 2, a person object is arranged in a coordinate space representing a person's mental state in accordance with an estimated state value related to emotions, and changes in the meeting's emotions are included. You can look back on the contents. However, it does not make it easy to see the display of the content of the meeting itself.

  The present invention has been made in view of the above circumstances, and when recording a meeting and displaying the contents of the meeting at a later date, the meeting information display that displays the contents of the meeting based on the recording in an easy-to-view manner for the user A system, a conference information display method, and a conference information display program are provided.

This invention
(I) a storage device that stores at least one of a document and an image of a document and a voice and an image of a conference participant as conference information, and a display device that displays at least one of the document and the image among the stored conference information A storage processing unit that associates each time zone obtained by dividing the conference period into a plurality of time zones and stores the conference information in the storage device, and stores the time zone of the conference based on the stored conference information. A calculation processing unit that calculates the degree of ascent, and a display that displays the stored conference information as a conference information object on the display device in a time-series arrangement with a degree of aggregation corresponding to the degree of climax in each time zone A conference information display system including a processing unit is provided.

Also, from a different point of view, this invention
(II) storing at least one of a document and an image of a document and a voice and an image of a conference participant in a storage device as conference information in association with each time zone obtained by dividing the entire period of the conference into a plurality of time zones; A step of calculating the degree of climax in each time zone of the conference based on the stored conference information, and arranging the stored conference information in time series as a conference information object, and according to the degree of climax in each time zone Providing a conference information display method including a step of displaying on a display device with different degrees of aggregation.
Furthermore, this invention
(III) receiving information sequentially input over a certain period in time series, and storing the received information in a storage device in association with each time period obtained by dividing the period into a plurality of time periods; A step of collecting the information as an information object in a time series and displaying the information on a display device; a step of setting an indicator in the time zone in response to selection of any time zone by the user; and the indicator And a step of lowering the degree of aggregation of display of display objects for the set time period and displaying the information on the display device.

From a different point of view, this invention
(IV) A process of storing at least one of a document and an image of a document and a voice and an image of a conference participant in a storage device as conference information in association with each time zone obtained by dividing the entire period of the conference into a plurality of time zones. , Processing for calculating the degree of climax in each time zone of the conference based on the stored conference information, and arranging the stored conference information in time series as a conference information object, according to the degree of climax in each time zone A conference information display program is provided for causing a computer to execute a process of displaying on a display device with different degrees of aggregation.

  The conference information display system according to the present invention arranges at least one of sentences and images in conference information stored in the display processing unit in time series as a conference information object, and the degree of aggregation according to the degree of climax in each time zone Since the contents of the conference are displayed on the display device at a later date, the contents of the conference based on the recording are displayed in an easy-to-view manner for the user.

It is explanatory drawing which shows the structural example of the meeting information display system in embodiment of this invention. It is a block diagram which shows the structural example of the display apparatus and server shown in FIG. In embodiment of this invention, it is explanatory drawing which shows an example of the screen in a meeting. In embodiment of this invention, it is explanatory drawing which shows the example of a display in the aggregated display mode of the called meeting. In embodiment of this invention, it is explanatory drawing which shows the example of balloon arrangement | positioning of a packing aggregation state and a loose aggregation state. It is explanatory drawing which shows the aspect of the meeting display screen different from the aggregation display mode shown in FIG. In embodiment of this invention, it is explanatory drawing which shows the data structure of the meeting information stored in a memory | storage part, and the example of a data table. 6 is a flowchart illustrating an example of processing executed by a control unit in a normal display mode in the embodiment of the present invention. In this embodiment of this invention, it is a flowchart which shows an example of the process which a control part performs by aggregate display mode. In FIG. 9, it is a flowchart which shows the process at the time of switching to a consolidated display mode. In this embodiment of this invention, it is a flowchart which shows the example of the process in which a control part produces | generates a climax data table. In this embodiment of this invention, it is a flowchart which shows the example of the process in which a control part performs a timeline division | segmentation. In embodiment of this invention, it is a graph which shows the example which determines the period which acquires each participant's gesture using control part audio | voice data. It is explanatory drawing which shows the modification of the meeting information display system 10 shown in FIG.

  Hereinafter, the present invention will be described in more detail with reference to the drawings. In addition, the following description is an illustration in all the points, Comprising: It should not be interpreted as limiting this invention.

≪Example of system for displaying conference information≫
(Embodiment 1)
FIG. 1 is an explanatory diagram showing a configuration example of the conference information display system of this embodiment. The conference information display system 10 shown in FIG. 1 includes display devices 13 a and 13 b and a server 11 which are arranged at conference bases L 1 and L 2, respectively, connected to a communication network 19. The server 11 may be in any of the bases, or may be in a place different from the bases L1 and L2.

The display device 13a installed at the base L1 is connected to a camera 15a that photographs the participant P1 and the participant P2. During the meeting, an image (live video) captured by the camera 15a at the site L1 is displayed on the display device 13b at the site L2. Similarly, during the conference, an image (live video) captured by the camera 15b at the site L2 is displayed on the display device 13a at the site L1.
Furthermore, a conference speaker phone 17a is connected to the display device 13a for collecting the speech of the participants P1 and P2 and amplifying the speech of the participant P3 at the base L2. During the conference, the speeches of the participants P1 and P2 collected by the speakerphone 17a at the base L1 are amplified from the conference speaker 17b at the base L2. Similarly, during the conference, the speech of the participant P3 picked up by the conference speaker phone 17b at the site L2 is amplified from the conference speaker phone 17a at the site L1.

  The server 11 stores image data captured by the cameras 15a and 15b during the conference and audio data collected by the conference speaker phones 17a and 17b. Furthermore, data of materials such as documents and images used in the meeting are stored.

FIG. 2 is a block diagram illustrating a configuration example of the display device 13a and the server 11 illustrated in FIG. FIG. 2A shows a configuration example of the display device 13a. The display device 13b has the same configuration. FIG. 2B is a configuration example of the server 11.
As shown in FIG. 2A, the display device 13a includes a control unit 21a, a storage unit 23a, a display unit 25a, an operation unit 27a, and a communication unit 29a. Further, in FIG. 1, a separate camera 15a and conference speakerphone 17a are connected.

  The controller 21a is composed of an input / output circuit and other peripheral circuits with the CPU at the center. The storage unit 23a includes a semiconductor memory such as a ROM and a RAM, a hard disk device, and the like. The display unit 25a includes a liquid crystal display device, a liquid crystal projector, and the like. The operation unit 27a receives power on / off of the display device 13a, a setting menu, and an operation on an object displayed on the display unit 25a, and is a transparent touch panel disposed on the front surface of the switch and the display unit 25a. The communication unit 29 a includes a communication interface with the communication network 19. Communication does not ask | require a communication system regardless of a radio | wireless and a wire communication. Further, not only the communication network 19 but also a communication interface with other peripheral devices (for example, Bluetooth (registered trademark), NIC, etc.) may be included.

  The camera 15a is preferably capable of shooting a moving image, but is not limited thereto, and may be a camera that shoots a still image. You can substitute a live image by continuously shooting still images and sending them frame by frame. The conference speakerphone 17a is a unit in which a microphone and a speaker are integrated so as to be convenient when a large number of people hold a meeting. However, the microphone and the speaker are not limited to this, and a headset for one person may be used. But you can.

As shown in FIG. 2B, the server 11 includes a control unit 21s, a storage unit 23s, and a communication unit 29s. Furthermore, a display unit 25s and an operation unit 27s may be provided. The controller 21s is composed of an input / output circuit and other peripheral circuits with the CPU at the center. The storage unit 23s includes a semiconductor memory such as a ROM and a RAM, a hard disk device, and the like. The display unit 25s is composed of a liquid crystal display device. The operation unit 27s receives power on / off of the server 11, a setting menu, and an operation on an object displayed on the display unit 25s. Specific examples include a keyboard and a mouse. The communication unit 29 s is a communication interface with the communication network 19. Communication does not ask | require a communication system regardless of a radio | wireless and a wire communication.
As shown in FIG. 2, the display device 13 a includes a control unit 21 a, a storage unit 23 a, and a communication unit 29 a similar to the configuration of the server 11, and thus can also function as the server 11.

(Embodiment 2)
FIG. 14 is an explanatory diagram showing a modification of the conference information display system 10 shown in FIG. A display device having a larger number of bases than that in FIG. 1 and not in FIG. 1 is shown. That is, at the base L3, a projection type data projector is used as the display device 13c. Regardless of the method of the data projector, a liquid crystal projector or a DLP (registered trademark) projector may be used. A camera 15c and a conference speakerphone 17c are connected to the data projector.
The display device 13d at the base L4 is a tablet terminal. The tablet terminal has a built-in camera, speaker, and microphone, so there is no need for external connection. The tablet terminal is connected to the communication network 19 via a wireless access point 18d.

(Embodiment 3)
The display devices 13 a and 15 b as shown in FIG. 1 are not limited to the configuration in which the display devices 13 a and 15 b are connected to the server 11 via the communication network 19, and either the display device 13 a or 15 b may have the function of the server 11. .
According to this aspect, the conference information display system 10 can be implemented without installing a specific server 11.
That is, the conference information display system 10 of the present invention includes a configuration in which the server 11 and the display device 13a or 15b are not physically separate but integrated.

≪Meeting display screen during meeting≫
FIG. 3 is an explanatory diagram illustrating an example of a screen during a conference displayed on the display unit 25a of the display device 13a.
In FIG. 3, the conference display screen 31 is displayed in a window on the display unit 25a. The conference display screen 31 is roughly divided into a participant moving image area 33, a balloon display area 35, and an operation menu 37.

  The participant video area 33 occupies about 2/3 of the area on the left side of the conference display screen 31, and a live video of the participant P3 captured by the camera 15b at the base L2 is displayed. In the upper part, the name “Nakajima” of the participant P3 of the base L2 is displayed. As the participant name, for example, an ID used for a login operation when a conference is held is displayed. Alternatively, the login ID of the participant and the face photograph are registered in the database in advance, and the face recognition technology is applied to the image of the camera 15b to display the names of the meeting participants who match the camera image. Also good.

The balloon display area 35 occupies about one third of the area on the right side of the conference display screen 31. The speeches of the conference participants are converted into text by voice recognition and sequentially displayed in a balloon shape. The speech balloons 45 and 47 correspond to this. A participant's face image icon (participant icon) is displayed in a small size in the upper left of each balloon, and it is possible to distinguish which participant is speaking. That is, the participant icon 44 is displayed on the upper left of the slide balloon 43. A participant icon 46 is displayed at the upper left of the speech balloon 45, and a participant icon 48 is displayed at the upper left of the speech balloon 47.
The background color of the balloon may be changed for each participant. It is possible to determine which participant is responsible for the speech as follows. Participant's login ID and voice characteristics are pre-associated and registered in the database. The voice recognition technology is applied to the voice data collected by the conference speakerphone 17a to search for a meeting participant who matches the voice data, and the face image icon of the person is added to the speech balloon. As the image used for the face image icon, a face photograph registered in advance in the database in association with the login ID of the participant is used.

In addition to the speech balloons 45 and 47 shown in FIG. 3, as speech balloons, documents and image data prepared by participants as materials can be displayed. A slide balloon 43 is shown in FIG. The participant can designate the material registered in advance in the server 11 at the timing used for explanation and display it in the balloon display area 35.
The slide balloon 43 and the speech balloons 45 and 47 are added from the top in chronological order from the top, and the old one moves downward and moves out of the screen. When the touch panel as the operation unit 27a is operated, the control unit 21a detects a screen swipe operation by the user. In response to the operation, the control unit 21a scrolls the balloon in the balloon display area 35.

  The operation menu 37 is an area for receiving user instructions regarding conference settings and progress. The operation menu 37 of FIG. 3 includes a “connection destination designation” button, a “file attachment” button (to perform material registration operation) related to the conference setting, a “MTG start” button for instructing the start of the conference, An “MTG end” button for instructing the end is displayed, but a button hidden by scrolling left and right is displayed. As a hidden button, there is a “call conference” button for calling a conference that has already ended and displaying it on the conference display screen 31. There is also a “call conference” button in a home menu (not shown) displayed after the display device 13a is activated. In FIG. 3, it is assumed that the user gives an instruction by touching each operation button of the operation menu 37. As a modification, a mode is also conceivable in which the control unit 21a recognizes a voice instruction input via the speakerphone 17a by applying voice recognition by the user giving a voice instruction.

≪Meeting display screen after the meeting≫
When the “call conference” button is touched and the conference to be called is designated, the control unit 21 a displays the content of the designated conference on the conference display screen 31.
The control unit 21a simply displays in time series when the conference is called, as in the conference (normal display mode). However, in the participant video area 33, a live video that is reproduced during the conference is displayed instead of a live video.
When a predetermined operation in the normal display mode (for example, an operation of swiping the screen up and down at a predetermined speed or higher) is performed, the mode is switched to the aggregate display mode. For example, in a normal display mode or during a conference, swiping the screen up in order to trace back to past topics, the past records of the currently ongoing conference are recalled and displayed in the aggregate display mode.
The content of the conference displayed on the conference display screen 31 in the aggregate display mode is different from the screen during the conference shown in FIG. Although there are aspects common to the screen during the meeting shown in FIG. 3 in that the balloons are displayed in chronological order, the arrangement of the balloons is displayed together to improve the listability, and each time zone is displayed. Is displayed with the degree of aggregation according to the degree of climax of the meeting.

FIG. 4 is an explanatory diagram showing a display example in the aggregated display mode of the called conference in this embodiment.
As shown in FIG. 4, a time axis indicating a transition of time in a predetermined increment is displayed at the left end of the conference display screen 31. The time axis extends in the vertical direction of the screen, and approaches from the past to the present as it goes from below to above. However, this is only an example, and the vertical and horizontal directions and the vertical direction are not limited as long as they are time series.

  In the example of FIG. 4, the time axis is in increments of 10 minutes. One time step corresponds to one time zone. In FIG. 4, four time zones 64a, 64b, 64c and 64d are displayed. On the time axis, the representative time of each time zone is displayed. The representative time of the time zone 64a is “10:30”, and the representative time of the time zone 64b is “10:40”. The representative time of the time zone 64c is “10:50”, and the representative time of the time zone 64d is “11:00”. The representative time is uniformly determined as the start time, intermediate time, or end time of each time zone. The user may be able to select one of them.

Since the user selects the slide balloon 51 by a touch operation and adds a mark indicating that it is important, the time zone 64c is displayed enlarged. The slide balloon 43 that belongs to the same time zone but is not selected by the user and the slide balloon 54 below the slide balloon 43 are displayed in an enlarged manner.
In addition, the time zone 64c is displayed in a loosely aggregated state, that is, in a state where the degree of aggregation is loose (low), because the degree of the conference climax exceeds the climax threshold. In the time zone including the conference information object to which the important mark is attached, the loosely aggregated state may be displayed as in the case where the conference climax degree exceeds the climax threshold. Also regarding the time axis, the display mode of the time axis is different from other time zones below the rising threshold.

The utterances and referenced materials in each time zone are displayed as balloons and displayed in a state of being gathered and arranged in the horizontal direction.
The conference information objects belonging to the time zone 64a are displayed in an aggregated display area 62. Similarly, the conference information objects belonging to the time zone 64b are displayed together in the packed display area 57. On the other hand, the conference information objects belonging to the time zone 64c including the important points are displayed in the loosely aggregated display area 55 in a distributed manner.
By collecting and displaying balloons for each time period, the user can easily look back and understand the progress of the meeting.

A “return” button 63 is arranged in the lower right corner of the conference display screen 31. When the “return” button 63 is touched, the control unit 21a returns the screen to the previous display content. That is, when the normal display mode is changed to the aggregate display mode, the normal display mode is restored. When transitioning to the aggregate display mode from the screen during the conference, the screen returns to the screen during the conference. When transitioning to the aggregate display mode from the “call conference” button in the home menu, the screen returns to the home menu.
In the aggregated display mode, the control unit 21a collects speech balloons for each time zone, aggregates them in the time axis direction, and superimposes and displays speech balloons belonging to the same time zone and arranged in the horizontal direction. As a result, a plurality of balloons are aggregated in the vertical time axis direction and in the horizontal direction in the same time zone, thereby improving the screen display listability.

Furthermore, the conference information display system 10 of this embodiment has a function of calculating the importance level of a statement, in other words, the degree of excitement of the conference. In the aggregate display mode, the speech balloons are aggregated for each time zone, and the degree of climax in each time zone is calculated and displayed as a climax degree graph superimposed on the screen (for example, see the climax degree graph 56 in FIG. 4). .
For a time zone in which the degree of climax exceeds a predetermined climax threshold, that is, a time zone including a statement having an importance level exceeding the climax threshold, the aggregation form of the balloons is different from the time zone below the climax threshold. Display. A plurality of pieces of information are displayed in an arrangement extending in the horizontal direction of the screen. As a result, the details of the conference are displayed more easily in the time zone in which the important content of the conference is discussed and the remarks of the importance level exceeding the climax threshold are made.

A specific attribute may be given to the speech balloon by performing a predetermined operation such as touching the speech balloon displayed by the participant. For example, when a gesture operation that draws a circle is performed on a speech bubble that the participant thinks is high in importance, the importance level of the speech exceeds a certain threshold value, that is, the importance level is high, that is, the importance level A high-quality attribute may be added. Here, a specific importance (high importance attribute) can be given not only to the speech balloon but also to the slide balloon (Embodiment 4).
In FIG. 4, a highly important attribute is given by performing an operation of drawing a circle on the slide balloon 51, and a circle frame is given and displayed so that the attribute can be visually recognized.

  FIG. 5A shows an example of the arrangement of balloons in a packed and aggregated state corresponding to a case where the degree of climax is equal to or less than the climax threshold. The center position of the balloon is shown in black with numerals 1 to 4 with white circles. The numbers indicate the order of the balloon positions, and 1 corresponds to the head of the order. As indicated by a sequence of numbers, balloons in the same time zone are arranged in time series from left to right. The interval between the center positions of adjacent balloons (balloon interval A) is about 70% of the diameter of the balloons, and the balloons are arranged so as to partially overlap.

On the other hand, as shown in FIG. 5B, in a range where the degree of climax is higher than the climax threshold (slowly aggregated state), the balloon interval B is set to be longer than A, and the balloons are arranged with less overlap and easy to see. Further, the balloon itself is displayed larger than usual.
Furthermore, an icon indicating a participant's video may be added in the loosely aggregated state (Embodiment 5). The user can play the participant's video by touching this icon.
In addition, balloons belonging to the same time zone may be displayed side by side in the horizontal direction, but a part of the balloons may be displayed in a time axis direction (Embodiment 6).

By changing the aggregation level of the display according to the meeting excitement level, the contents of the agenda are displayed in more detail during the time zone when the important contents were discussed and the meeting was excited. Furthermore, it arrange | positions so that operation for grasping | ascertaining the content of a meeting may be easy.
The speech balloon may be moved by animation when the normal display is changed to the aggregated display (the seventh embodiment).
As described above, for the time zone including the important points, the coordinate position of each balloon is set in the packed and aggregated state shown in FIG. For a time zone that does not include an important point, the coordinate position of each balloon is set in the loosely aggregated state shown in FIG.

  FIG. 6 is an explanatory view showing a mode of a conference display screen different from the aggregate display mode shown in FIG. 6 is different from that in FIG. 4 in a time zone 64b. FIG. 6 shows a state after the user swipes right to the balloon displayed in the packed state in the time zone 64b. When the touch panel as the operation unit 27a detects that the speech bubble of the time zone 64b has been swiped to the right, the display unit 21a shifts the speech speech bubble 47 that is displayed in a packed state and hidden under the speech speech bubble 45 to the right. Display the contents of the balloon that was hidden. As described above, the user can easily refer to the contents of the hidden speech balloon by performing a predetermined operation on the speech balloon hidden in the packed and aggregated state.

FIG. 7 shows an example of the data structure of the conference information object stored in the storage unit 23s of the server 11 and a data table for managing the data. However, the present invention is not limited to this, and some data may be stored in, for example, the storage unit 23a of the display device 13a and exchange data with the server 11 or other display devices.
Of the conference information objects stored in the storage unit 23 s, material data and audio data are displayed as balloons, and are therefore managed by the balloon data table 71. A balloon is a typical meeting information object. The balloon data table 71 has items of slide ID, timeline, user ID, text-to-speech data, related file name, and attribute data as balloon attributes.

  The audio data collected by the conference speakerphones 17 a and 17 b during the conference is divided for each utterance during the conference, and each utterance is registered in the balloon data table 71. Each speech balloon has a speech start time registered as a timeline. The speaker's ID is registered as the user ID. This is the ID of the participant searched using the voice recognition technology. As the text-formed voice data, the text data of the speech that has been converted into text using a voice recognition technique is registered. If the related file name is speech balloon data, there is no corresponding registration content. The attribute data indicates whether or not the user has added the mark as an important speech balloon.

  The moving image data 73 stores the moving images of the participants at each base in the compressed data format of still images. Since each still image is stored as a difference between the previous and next still images, the data capacity is reduced. A timeline (photographing time) is recorded on the still image frame. In the example of FIG. 7, the audio data does not exist as single data but is included in the moving image data 73. However, the voice data may be stored separately from the moving picture data 73 in order to facilitate generation of the text voice data and identification of the speaker of the speech.

In the user information table 75, the participant ID (user ID), the participant name (user name), and face photo data (user face photo) are registered before the conference. The user's face photo data is the source of the icon attached to the balloon. For example, when a conference participant logs in to the conference information display system 10 with his / her ID, the logged-in user is registered in the user information table 75.
The timeline delimiter data table 77 records breakpoints for setting time zones of balloons to be displayed in a consolidated manner and timelines (time) corresponding to the breakpoints. Balloons existing between the time of the breakpoint and the time of the next breakpoint are grouped and aggregated at the time of aggregation display. The timeline delimiter data table 77 is generated in parallel with the progress of the conference.

The time as the break point may be a predetermined time interval. For example, the time when the conversation of all the participants is interrupted may be extracted from the recorded audio data and set as the break point. . More specifically, the time when the total volume of all the participants is equal to or lower than a predetermined value for a certain time is extracted as a breakpoint.
Alternatively, a specific keyword may be detected as a topic break. For example, keywords such as “I will explain from now on”, “Next agenda is”, and “My opinion is” are registered in advance, and the speech corresponding to the keyword using voice recognition technology Extract the time of day as a breakpoint.

When displaying in the integrated display mode, the balloon is displayed by referring to the timeline delimiter data table 77 and checking which delimiter the timeline of each balloon registered in the balloon data table 71 belongs to. The time zone can be determined.
Thus, by referring to the balloon data table 71 and the timeline delimiter data table 77, the association between the balloon as the conference information object and the time zone can be examined.

With respect to the moving image data 73, the timeline attached to the moving image data 73 can be checked with respect to which breakpoint it belongs by referring to the timeline delimiter data table 77. ) Can be determined. In this way, it is possible to check the association between the moving image as the conference information object and the time zone.
The climax data table 79 stores the time associated with the conference information object whose importance is calculated and the importance of the conference information object. The importance is calculated using a predetermined method. Furthermore, a flag (important point flag) indicating whether or not the degree of importance has exceeded a predetermined climax threshold is stored. In other words, a conference information object having an importance level exceeding the climax threshold is marked as an important point. The climax data table 79 is generated in parallel with the progress of the conference.

≪Processing procedure≫
The above process will be described using a flowchart. In the following description, the balloon data table 71, the moving image data 73, the user information table 75, the timeline delimiter data table 77, and the climax data table 79 are collected and stored in the storage unit 23s of the server 11 for simplification. It is supposed to be.
In the conference information display system 10, the control unit 21s of the server 11 and the control unit of the display device at each base (for example, the control unit 21a of the display device 13a at the base L1) perform data and display processing. Therefore, the storage processing unit, the calculation processing unit, and the display processing unit may be configured to be processed by any of the control units 21s and 21a, or may be configured to be processed in cooperation between the two companies. For simplification, in the following description, the control unit 21s of the server 11 controls the system, and the control unit of the display device (for example, the control unit 21a of the display device 13a of the base L1) is simply controlled by limbs. Assume that the processing of the unit 21s is assisted.

FIG. 8 is a flowchart illustrating an example of processing executed by the control unit 21s in the normal display mode. The normal display mode corresponds to the display form of FIG. This mode is during a meeting or immediately after the “call meeting” button is pressed.
The control unit 21s repeatedly executes the process illustrated in FIG. In FIG. 8, the process is shown by a loop in the flowchart, but the control unit 21 s executes multitask processing in parallel with other processes that are not actually described in FIG. 8.

In FIG. 8, the control unit 21s checks whether or not the conference speakerphone has detected the voice of the participant (step S11). When speech is detected (Yes in step S11), speech recognition processing is applied to the speech (step S13) to generate textified speech data, and the generated text data is stored in the balloon data table 71 (step S13). S15). Then, the voice data converted into text is displayed as a speech balloon on the conference display screen 31 (step S17).
Further, the control unit 21s checks whether or not the user has performed an operation for requesting a slide balloon (step S19). If there is a request (Yes in step S19), the designated file is read (step S21) and stored in the balloon data table 71 (step S23). Then, the read file is displayed on the conference display screen 31 as a slide balloon (step S25).

After the balloon display processing, the control unit 21s checks whether or not the operation unit 27a has detected an operation on the conference display screen 31 by the user (step S29). When the operation is performed, it is checked whether or not the operation is a request for enlarging and displaying the balloon (step S31). In this example, the operation of tapping the displayed balloon is an operation for requesting enlarged display. When the operation is performed (Yes in step S31), the tapped balloon is enlarged and displayed (step S33).
When the swipe operation is performed in the vertical direction (step S35), the speed of the swipe is determined (step S37). If the swipe speed is equal to or less than a predetermined swipe speed threshold (No in step S37), scroll processing is performed. On the other hand, if the swipe speed exceeds the swipe speed threshold, the display is switched to the aggregate display mode (step S39). The process of switching to the aggregate display mode will be described with reference to FIG.

Furthermore, the control unit 21s checks whether or not a specific operation (an operation that encircles in this example) is performed on the blowout (step S43). When an operation of encircling is performed, it is a user's intention indication that it is an important content, so a cooperation flag is set as attribute data (see the balloon data table 71 in FIG. 7). Then, the speech balloon targeted for operation is highlighted and displayed (step S45; see slide speech balloon 51 in FIG. 4).
The above is the processing in the normal display mode.

Next, the switching process to the consolidated display mode in step S39 will be described.
FIG. 9 is a flowchart illustrating an example of processing executed by the control unit 21s in the aggregate display processing.
First, the control unit 21s executes a process of switching the conference display screen 31 to the aggregate display mode (step S51). This process will be described in detail with reference to FIG.
Thereafter, the control unit 21s checks whether or not the operation unit 27a has detected an operation on an object displayed on the conference display screen in the aggregate display mode (step S53). If any operation is detected (Yes in step S53), it is checked whether or not the operation is a vertical swipe operation (step S55).

If the swipe operation is in the vertical direction (Yes in step S55), the process of vertical scroll display is performed (step S57).
If it is a touch operation on the speech balloon (step S59). The touched balloon is enlarged and displayed. If the operation is a touch operation on a moving image object, the moving image is reproduced (step S61).
Alternatively, if the operation is to swipe the balloon in the horizontal direction (step S63, see FIG. 6), the balloon is expanded and displayed in the horizontal direction (step S65).
If it is a touch operation on the “return” button 63 (step S67), the display is returned to the previous stage (step S69).

Next, details of step S51 will be described.
In FIG. 9, it is a flowchart which shows the process at the time of switching to a consolidated display mode.
As illustrated in FIG. 10, the control unit 21 s displays the degree of climax in each time zone calculated by a climax degree calculation process described later on the conference display screen 31 (step S <b> 71). Further, referring to the timeline delimiter data table 77 (step S73), the moving image data is divided into each time zone, a balloon is assigned to each time zone, and a conference information object to be displayed is determined for each time zone (step S73). S75). That is, the conference information objects to be displayed on the conference display screen 31 are grouped by time zone.

Further, for each time zone, it is checked whether or not a conference information object (important point) having an importance level exceeding the climax threshold is included (step S77).
For the time zone including the important point (Yes in step S77), the arrangement of the conference information objects is determined in a loosely aggregated state (step S79). The arrangement in the loosely aggregated state corresponds to FIG. For a time zone that does not include an important point (No in step S77), the arrangement of the conference information objects is determined in a packed and aggregated state (step S81). The packed and consolidated state corresponds to FIG.

And according to the determined arrangement | positioning, the meeting object which belongs to each time slot | zone is displayed on the meeting display screen 31 (step S83). At this time, each conference object may be gradually moved from the position of the normal display mode to display an animation.
The above is the processing when switching to the aggregate display mode.

FIG. 11 is a flowchart illustrating an example of processing in which the control unit generates a climax data table in this embodiment. The control unit 21s executes the process shown in FIG. 11 and the process shown in FIG. 8 in parallel.
As shown in FIG. 11, while the conference is in progress, the control unit 21s waits for any participant to start speaking (step S101), resets the timer (step S103), and starts a predetermined period from that point. The following monitoring is executed until (in this example, 20 seconds) elapses (step S109). That is, the participant's live video is analyzed, and the number of participants who have performed a pointing operation during the period is counted (step S105), and the number of participants who have performed a nodding operation is counted (step S105). S107).
Then, the importance level of the utterance is calculated based on the number of persons who performed the pointing action and the nodding action during the period (step S111). The calculation method will be described later.

If the calculated importance level exceeds the climax threshold (step S113), an important point flag is set for the statement (step S115). For the setting of the important point flag, refer to the balloon data table 71 in FIG.
In this way, each time a statement is made, the importance level of that statement is calculated and recorded in the climax data table 79 in association with the statement start time (step S117).
In FIG. 11, the control unit 21 s of the server 11 calculates the climax degree, but this is only an example. For example, the control unit 21a of the display device 13a calculates the degree of climax related to the base L1, and similarly, the display device 13b calculates the degree of climax related to the base L2, and the control unit 21s of the server 11 comprehensively calculates them. A typical climax degree may be calculated.

A supplementary explanation will be given for the processing of FIG.
In order to use for calculation of the importance level of the utterance, the control unit 21s acquires the physical reaction (gesture) of each participant within 20 seconds after the utterance of any participant starts. Specifically, a known image recognition technique is applied to the images of the participants taken by the cameras 15a and 15b to extract the pointing action and the nodding action of each participant. Then, the degree of excitement of the conference is calculated using the presence or absence of operation as a parameter.

  The start of speech is determined based on the volume of the audio data. FIG. 13 is a graph showing an example of determining the start time of each participant using speech data of each participant and determining a period for acquiring each participant's gesture in this embodiment. The conversation volume of the participants P1, P2 and P3 corresponds to the volume picked up by each microphone when the microphone corresponding to each participant is placed. A microphone shared by a plurality of participants, such as the conference speakerphone 17a of FIG. 1, extracts the speech of each participant using a speech recognition technology, and the volume of the extracted speech corresponds to the graph of FIG.

The pointing action is obtained from the movement of the participant's hand. The nodding motion is acquired from the facial motion (face recognition) of the participant.
Based on the number of meeting participants who performed the pointing action and the number of persons who performed the nodding action and indicated consent, the importance of the speech is calculated. For example, the importance is calculated based on the following formula.

Importance = (Percentage of all participants who acted during the 20-second period after remarks) x (Percentage of all participants who noded during that period)

  According to the above formula, the value of the previous item is 1 when all the participants perform a pointing operation, and the value of the previous item is 0 when no one performs the pointing operation. Further, the value of the latter term is 1 when all participants perform a nodding operation, and the value of the latter term is 0 when no one has performed a nodding operation. Therefore, the importance value is 1 when all perform a pointing action and a nodding action, and the importance level is 0 when no one has performed a pointing action or no one has performed a nodding action. become.

The control unit 21s generates a graph of the degree of excitement by associating the utterance with the importance calculated, the timeline when the utterance is registered in the balloon data table 71, that is, the utterance start time with the importance. In order to make the graph easy to see, the importance of each message may be displayed by approximating the curve. The control unit 21s refers to the timeline delimiter data table 77 and determines which time zone it belongs to. A time zone including a statement in which the calculated importance degree exceeds the climax threshold is determined as a time zone in which an important statement has occurred. Then, as shown in the time zone 64c of FIG. 4, the time axis expression and the degree of balloon aggregation are displayed in a manner different from the time zone of low importance.
As described above, the control unit 21 s calculates the importance level of the utterance by monitoring the action indicated by each participant at the beginning of the utterance during the conference, and adds the importance level (rising degree) to the climax data table. We add sequentially.

Next, a process for generating the timeline delimiter data table 77 will be described. The timeline break may be a predetermined time interval (for example, 10 minutes) or may be determined according to the progress of the conference as shown in FIG.
The control unit 21s executes the process shown in FIG. 12 and the processes shown in FIGS. 8 and 11 in parallel.

As shown in FIG. 12, the control unit 21s obtains a total speech volume obtained by summing up the voices of the participants in order to find a timeline break during the conference (step S121). It is monitored whether or not the volume is below a predetermined volume over a predetermined period, that is, whether there is a silent period (step S123). If there is a silent period (Yes in step S123), the routine proceeds to step S127, and adds a break point to the timeline break data table 77.
Further, the control unit 21s analyzes the participant's voice data and monitors whether or not a specific keyword registered in advance is included in the utterance (step S125). If a keyword is included in the utterance (Yes in step S125), the routine proceeds to step S127.

In step S127, the control unit 21s adds a new break point to the timeline break data table 77 together with the start time of the message (step S127).
In order to make the breakpoint value distinguishable from the previous breakpoint, in this example, a value obtained by adding 1 to the previous breakpoint value is registered in the timeline breakpoint data table 77 as a new breakpoint value. (Step S129).
As described above, the control unit 21s sequentially adds break points to the timeline break data table 77 during the progress of the conference.

As mentioned above,
(I) A conference information display system according to the present invention includes a storage device that stores at least one of a document and an image of a material and a voice and an image of a conference participant as conference information, and a document and an image of the stored conference information. A storage processing unit that includes a display device that displays at least one of them, divides a conference period into a plurality of time zones, and associates the conference information with the conference information, and stores the conference information in the storage device; A calculation processing unit that calculates the degree of climax in each time zone of the conference based on the stored conference information as a conference information object, and a degree of aggregation according to the climax degree in each time zone and a time series A display processing unit configured to display the display device on the display device.

In the present invention, the storage device stores data stored as conference information. A specific aspect thereof is, for example, a nonvolatile storage device such as a hard disk device or a flash memory. In the above-described embodiment, the storage device is included in the configuration of the server 11.
The storage device and the display device are essentially separate devices, but may be housed in a single housing.
The storage processing unit, calculation processing unit, and display processing unit may use dedicated hardware, but their functions may be realized by a computer executing a program.

The storage processing unit, the calculation processing unit, and the display processing unit may be arranged in the storage device or another device connected to the display device, for example, an information processing device. Or you may arrange | position in either of the said display apparatuses.
Documents as conference information include various types of documents (images created using document creation software and form calculation software, image data created using drawing software, etc.). May be included). The voice of the conference participant is voice data recorded using a microphone. The image of the conference participant is image data taken using a camera, and may be a still image or a moving image.

Furthermore, the time zone may be a predetermined period (for example, every 10 minutes), but may be adaptively determined at the time of recording or after recording. For example, it may be determined that a new time zone is started when the speech is interrupted for a predetermined period.
The degree of excitement is an index related to the importance of the content of the meeting in each time zone. Of course, when you look at the meeting records at a later date, you should naturally prioritize the important parts. However, it is difficult for a person who understands the content and background of the meeting to determine whether it is important. Therefore, an index that can be extracted from the stored conference information and is considered to have a sufficiently close correlation with the importance degree is expressed as a degree of climax.

  For example, it is presumed that there is a high possibility that important contents are discussed during a time period in which participants frequently point to materials or nod during the meeting. The pointing action and the nodding action can be extracted by using image recognition technology if there is moving image data that captures the state of the participant. In addition, it is speculated that there is a lively discussion on important contents and that participants tend to speak loudly. If there is audio data in which a participant's utterance is recorded, the volume of the utterance can be extracted using an audio processing technique.

Furthermore, the preferable aspect of this invention is demonstrated.
(Ii) The conference information includes a moving image of a participant in the conference, and the display processing unit can operate the moving image in the time zone in a time zone in which the degree of climax is higher than a predetermined threshold. The icon may be displayed as one meeting information object.
In this way, icons that can be used to operate videos during that time period will be displayed for the time period when important content was discussed and the meeting was exciting, so the user can easily operate the video during that time period. Can be viewed.

(Iii) The display processing unit, when any one of the conference information objects that are aggregated and displayed with a low degree of climax is selected by the user, the selected conference information object and the conference information aggregated together Objects may be distributed and displayed on the display device.
In this way, even for conference information objects that are aggregated and displayed with a low degree of climax, the contents of the selected conference information objects can be displayed in an easy-to-read manner for the user by being distributed according to the user's selection. Can do.

  (Iv) The conference information includes a moving image of a participant in the conference, and the calculation processing unit is configured to perform the pointing operation and the nodding operation of the participant extracted from the moving image in each time period. The degree of swell may be calculated.

(V) The display processing unit displays the conference information objects in a time zone in which the degree of climax is high compared to a time zone in which the climax degree is low, and is distributed and displayed. Also good.
In this way, the conference information object is displayed at a low degree of aggregation during the time when the conference was excited because important content was discussed, and the content of the conference based on the record is displayed in an easy-to-view manner for the user.

(Vi) The display processing unit may display the size of the conference information object larger in the time zone where the degree of climax is higher than in the time zone where the climax degree is low.
In this way, the conference information object is displayed in a large size for the time zone in which important content is discussed and the conference is excited, and the content of the conference based on the record is displayed in an easy-to-view manner for the user.
(Vii) The display processing unit may represent the degree of climax in each time zone as a graph and display the degree on the display device together with the conference information object.
In this way, the user can preferentially view the record by inferring a time zone in which important contents were discussed and the meeting was excited with reference to the degree of climax displayed in the graph.
Preferred embodiments of the present invention include combinations of any of the plurality of embodiments described above.
In addition to the embodiments described above, there can be various modifications of the present invention. These modifications should not be construed as not belonging to the scope of the present invention. The present invention should include the meaning equivalent to the scope of the claims and all modifications within the scope.

10: Conference information display system, 11: Server, 13a, 13b, 13c, 13d: Display device 15b, 15c: Camera, 17a, 17b, 17c: Conference speakerphone, 18d: Wireless access point, 19: Communication network 21a, 21s: control unit, 23a, 23s: storage unit, 25a, 25s: display unit, 27a, 27s: operation unit, 29a, 29s: communication unit 31: conference display screen, 33: participant video area, 35: balloon display area 37: Operation menu, 43, 51, 54: Slide balloon, 44, 46, 48, 50, 52, 59, 61: Participant icon, 45, 47, 49, 58, 60: Speech balloon, 55: Loose aggregation Display area, 56: climax graph, 57, 62: packed aggregate display area, 63: “return” button, 64 a, 64b, 64c, 64d: time zone 71: balloon data table, 73: moving image data, 75: user information table, 77: timeline delimiter data table, 79: rising data table L1, L2, L3, L4: base P1, P2, P3, P4, P5: Participants

Claims (10)

A storage device that stores at least one of a document and an image of a document and a voice and an image of a conference participant as conference information;
Including a display device that displays at least one of a document and an image of stored conference information,
A storage processing unit that associates each time zone obtained by dividing a conference period into a plurality of time zones and stores the conference information in the storage device;
A calculation processing unit that calculates the degree of climax in each time zone of the conference based on the stored conference information;
A conference information display system comprising: a display processing unit configured to display the stored conference information as a conference information object on the display device in a time-series arrangement with a degree of aggregation corresponding to the degree of climax in each time zone. The conference information includes a video of the conference participants,
The conference according to claim 1, wherein the display processing unit displays, as a conference information object, an icon capable of operating a moving image in the time zone in a time zone where the degree of climax is higher than a predetermined climax threshold. Information display system.   The display processing unit distributes the selected conference information object and the conference information object that are aggregated together when any of the conference information objects that are aggregated and displayed with a low climax is selected by the user. The conference information display system according to claim 1, wherein the conference information is displayed on the display device. The conference information includes a video of the conference participants,
The said calculation process part calculates the said excitement degree based on the frequency of the participant's pointing operation and nodding operation extracted from the said moving image in each time slot | zone. Conference information display system.   The display processing unit displays a time zone in which the degree of climax is high in a distributed manner with a lower degree of aggregation for displaying the conference information objects than a time zone in which the degree of climax is low. 4. The conference information display system according to any one of 4.   The said display process part displays the magnitude | size of a meeting information object larger in the time slot | zone where the said climax degree is high compared with the time slot | zone where the said climax degree is low. Meeting information display system.   The conference information display system according to claim 1, wherein the display processing unit represents the degree of climax in each time zone as a graph and displays the degree on the display device together with the conference information object. Storing at least one of a document and an image of a document and a voice and an image of a conference participant in a storage device as conference information in association with each time zone obtained by dividing the entire period of the conference into a plurality of time zones;
Calculating the degree of climax in each time zone of the conference based on the stored conference information;
A conference information display method comprising: arranging stored conference information in a time series as a conference information object, and displaying the conference information on a display device with different degrees of aggregation according to the degree of climax in each time zone. A process of storing at least one of a document and an image of a document and a voice and an image of a conference participant in a storage device as conference information in association with each time zone obtained by dividing the entire period of the conference into a plurality of time zones;
Processing for calculating the degree of climax in each time zone of the conference based on the stored conference information;
Conference information display for causing a computer to execute a process of arranging stored conference information in a time series as a conference information object, and displaying on a display device with a different degree of aggregation according to the degree of climax in each time zone program. Receiving information input sequentially over a period of time in time series;
Storing the received information in a storage device in association with each time period obtained by dividing the period into a plurality of time periods;
A step of collecting the information as an information object in time series and displaying it on a display device;
Responsive to user selection of any time zone and setting an indicator for that time zone;
An information display method comprising: displaying on the display device at a reduced degree of aggregation of display of display objects for a time zone in which the index is set.