JP7061860B2 - Digest data generation device, digest data playback device, digest data generation system, digest data generation method and program - Google Patents

Description

The present invention relates to a digest data generator and the like.

Traditionally, a conference system has been used to record audio and video of a multi-person meeting such as a conference or training so that it can be looked back on later.

In order to confirm the conference data such as recorded audio data and video data, it is generally performed by sequentially playing back from the start of the conference. However, in some meetings, the participants of the meeting are talking about the discussion, while others are silent. Therefore, in order to efficiently check the recorded contents of a meeting, it is desirable to be able to identify an important scene in the meeting and check it only in that scene, such as when discussing a meeting.

Therefore, a technique for identifying a part considered to be important in a conference has been proposed. For example, when there is a statement that is judged to be important, or when there is a valid statement that should be confirmed later, there is an invention that can add an index to the statement that you want to play later by instructing the insertion of a bookmark. It is disclosed (see, for example, Patent Document 1).

In addition, the feature amount as a time series along the progress of the meeting is extracted from the video data, and based on the feature amount, the degree of excitement as a time series along the progress of the meeting is calculated as an absolute evaluation value and participates. An invention relating to a conference analyzer that estimates a relative degree of excitement in consideration of a person's character, place of participation, content of remarks, and conversational relationship is disclosed (see, for example, Patent Document 2).

Further, there is known a system that can store captured images in chronological order. For example, a still image is stored for each sampling time together with the state during the meeting as audio data. As a result, when transferring a meeting, it is possible to confirm the state of the meeting with an image as well as a voice.

Japanese Unexamined Patent Publication No. 2005-109928 Japanese Unexamined Patent Publication No. 2016-012216

Here, in order to make it easier to transfer the meeting, the method of directly instructing the scene that seems to be important in the meeting has the problem that the instruction may interfere with the conversation and that the instruction is too focused on the conversation and the instruction is forgotten. rice field.

Further, when storing the conference data, it is possible to memorize the detailed situation by taking a picture with a fine sampling time, but there is a problem that the amount of information becomes large and it becomes difficult to identify an important scene.

In particular, as the amount of information increases, the amount of storage of conference data increases. By doing so, there are problems that the free space of the storage area becomes insufficient and that a large communication band must be secured when receiving via the network.

An object of the present invention is to solve the above-mentioned problems, and the importance of content data including audio data and image data is determined for each predetermined time interval, and the image is determined according to the importance. It is to provide a device for generating digest data from which data has been deleted.

In order to solve the above-mentioned problems, the digest data generator of the present invention is used.
Image data acquisition means for acquiring image data and
A storage means for storing audio data and image data periodically acquired in response to audio data as content data,
A feature amount determining means for determining a feature amount for each section of a predetermined time length with respect to the content data,
An importance determination means for determining the importance of each section from the feature amount,
A digest data generation means for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
It is characterized by having.

The digest data generator of the present invention is
Image data acquisition means for acquiring image data and
A storage means for storing audio data and image data periodically acquired in response to audio data as content data,
With respect to the content data, a feature amount determining means for determining a feature amount for each section having a predetermined time length for each feature based on a plurality of different features,
An importance determination means for determining the importance of each section based on the feature amount of each section determined for each feature, and
A digest data generation means for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
It is characterized by having.

The digest data generation system of the present invention is
A digest data generation system in which a terminal device is connected to a server device.
The terminal device is
Image data acquisition means for acquiring image data and
A content data transmission means for transmitting audio data and image data periodically acquired corresponding to the audio data as content data to the server device, and
The server device is equipped with
A content data receiving means for receiving the content data from the terminal device,
A feature amount determining means for determining a feature amount for each section of a predetermined time length with respect to the content data,
An importance determination means for determining the importance of each section from the feature amount,
A digest data generation means for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
A digest data transmitting means for transmitting the digest data to the terminal device,
It is characterized by having.

The digest data generation system of the present invention is
A digest data generation system in which a terminal device is connected to a server device.
The terminal device is
Image data acquisition means for acquiring image data and
A content data transmission means for transmitting audio data and image data periodically acquired corresponding to the audio data as content data to the server device, and
The server device is equipped with
A content data receiving means for receiving the content data from the terminal device,
With respect to the content data, a feature amount determining means for determining a feature amount for each section having a predetermined time length for each feature based on a plurality of different features,
An importance determination means for determining the importance of each section based on the feature amount of each section determined for each feature, and
A digest data generation means for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
A digest data transmitting means for transmitting the digest data to the terminal device,
It is characterized by having.

The digest data generation method of the present invention is
Image data acquisition step to acquire image data and
A storage step that stores audio data and image data that is periodically acquired in response to audio data as content data,
A feature amount determination step for determining a feature amount for each section of a predetermined time length with respect to the content data, and a feature amount determination step.
An importance determination step for determining the importance of each section from the feature amount,
A digest data generation step for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
It is characterized by including.

The digest data generation method of the present invention is
Image data acquisition step to acquire image data and
A storage step that stores audio data and image data that is periodically acquired in response to audio data as content data,
With respect to the content data, a feature amount determination step for determining a feature amount for each section having a predetermined time length for each feature based on a plurality of different features, and a feature amount determination step.
An importance determination step for determining the importance of each section based on the feature amount for each section determined for each feature, and
A digest data generation step for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
It is characterized by including.

The program of the present invention
On the computer
Image data acquisition function to acquire image data and
A storage function that stores audio data and image data that is periodically acquired in response to audio data as content data,
A feature amount determination function that determines a feature amount for each section of a predetermined time length for the content data, and a feature amount determination function.
An importance determination function for determining the importance of each section from the feature amount,
A digest data generation function that generates digest data from which the image data is deleted according to the importance determined for each section from the content data.
Is characterized by executing.

The program of the present invention
On the computer
Image data acquisition function to acquire image data and
A storage function that stores audio data and image data that is periodically acquired in response to audio data as content data,
With respect to the content data, a feature amount determination function for determining a feature amount for each section of a predetermined time length for each feature based on a plurality of different features, and a feature amount determination function.
An importance determination function that determines the importance of each section based on the feature amount of each section determined for each feature, and
A digest data generation function that generates digest data from which the image data is deleted according to the importance determined for each section from the content data.
Is characterized by executing.

According to the present invention, content data including audio data and conference image data is stored, a feature amount is determined for each predetermined time interval of the content data, and the importance is determined from the feature amount. Further, the digest data is generated from the content data by deleting the image data included in the content data of the non-important section based on the importance. As a result, the amount of information to be stored changes according to the importance, so that it is possible to easily identify important scenes and to suppress the required storage capacity and communication band.

It is a figure for demonstrating the whole structure of the conference support system in 1st Embodiment. It is a figure for demonstrating the functional structure of the display device in 1st Embodiment. It is a figure which showed an example of the data structure of the conference data in 1st Embodiment. It is a figure which showed an example of the data structure of the evaluation information in 1st Embodiment. It is a figure which showed an example of the data structure of the bookmark information in 1st Embodiment. It is a figure which showed an example of the data structure of the importance table in 1st Embodiment. It is an operation flow for demonstrating the main process in 1st Embodiment. It is an operation flow for demonstrating the participant evaluation signal transmission processing in 1st Embodiment. It is a figure for demonstrating the operation example of the mobile terminal apparatus in 1st Embodiment. It is an operation flow for explaining the importance degree determination process in 1st Embodiment. It is a figure for demonstrating the operation example of the importance degree determination process in 1st Embodiment. It is an operation flow for demonstrating the optimization process in 1st Embodiment. It is a figure for demonstrating the operation example of the optimization process in 1st Embodiment. It is an operation flow for demonstrating the digest data reproduction processing in 1st Embodiment. It is a figure for demonstrating the operation example of the digest data reproduction processing in 1st Embodiment. It is an operation flow for demonstrating the digest data reproduction processing in 2nd Embodiment. It is an operation flow for demonstrating the reproduction section narrowing process in 2nd Embodiment. It is a figure for demonstrating the operation example of the reproduction section extraction in 2nd Embodiment. It is a figure for demonstrating the operation example of the digest data reproduction processing in 2nd Embodiment. It is a figure for demonstrating the operation example of the digest data reproduction processing in 2nd Embodiment. It is an operation flow for explaining the importance degree determination process in 3rd Embodiment. It is a figure for demonstrating the operation example of the importance degree determination process in 3rd Embodiment. It is a figure for demonstrating the operation example in the case of using a plurality of feature quantities in 3rd Embodiment. It is a figure for demonstrating the operation example of the digest data reproduction processing in 3rd Embodiment. It is a figure for demonstrating the operation example of the digest data reproduction processing in 4th Embodiment. It is a figure for demonstrating the whole structure of the conference support system in the 6th Embodiment. It is a figure for demonstrating the whole structure of the conference support system in 8th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In this embodiment, as an example, a conference support system including a display device having a digest data generation device according to the present invention will be described.

[1. First Embodiment]
[1.1 Overall configuration]
First, the overall configuration of the conference support system 1 of the present embodiment will be described with reference to FIG. As shown in FIG. 1, in the conference support system 1, a mobile terminal device 20 is connected to a display device 10 via a NW (network) and an AP (access point). One mobile terminal device 20 may be connected, or a plurality of mobile terminal devices 20 may be connected. Further, in FIG. 1, the mobile terminal device 20 is shown to be connected to the display device 10 by the same network, but may be connected via another network. Further, it may be connected by wire.

Further, one or a plurality of button-type beacon devices 30 may be connected to the display device 10. In FIG. 1, the button-type beacon device 30 is displayed as being connected to the display device 10 by a short-distance communication means, but may be connected by a network. Further, it may be connected by wire as a button type input device.

[1.2 Function configuration]
The functional configuration of the display device 10 will be described with reference to FIG. The display device 10 includes a control unit 110, a storage unit 120, an audio input / output unit 140, a video input unit 150, a communication unit 160, an operation unit 170, and a display unit 180.

The control unit 110 is a functional unit for controlling the entire display device 10. The control unit 110 realizes various functions by reading and executing various programs stored in the storage unit 120, and is configured by, for example, a CPU (Central Processing Unit) or the like.

The storage unit 120 is a functional unit that stores various programs and various data necessary for the operation of the display device 10. The storage unit 120 is composed of, for example, a semiconductor memory, an HDD (Hard Disk Drive), or the like.

Here, the storage unit 120 stores the conference data 122, the evaluation information 132, the bookmark information 134, the importance table 136, and the digest data 138. Here, the conference data 122 stores basic attributes related to the conference and content data recording the conference for each conference. In this embodiment, the basic attributes related to the conference are stored as the basic information 124 of the conference. Further, as the content data, the audio data 126, the image pickup data 128, and the screen data 130 are stored. In the present embodiment, the content data will be described as storing audio data and image data, but moving image data and other data related to the conference may be included.

The data stored in the conference data 122 will be described with reference to FIG. The basic information 124 is basic information about the conference. For example, the meeting name (for example, "20170403_regular meeting 1"), the start date and time of the meeting (for example, "April 3, 2017 10:00: 00"), and the meeting participants (for example, "Participant A, Participant B"). , Participant C ”), and other basic information about the conference is stored. In addition to this, information about the conference such as the conference location and the agenda may be stored.

The voice data 126 is obtained by inputting voice to the voice input / output unit 140 and storing the output voice data. The audio data may be in a format that can be reproduced and output by the display device 10, and is, for example, information encoded by an audio encoder such as MP3 (MPEG Audio Layer-3), AAC (Advanced Audio Coding), or WAV. It is common to have.

The image pickup data 128 is a storage of image data taken and output by the video input unit 150, and is, for example, still image data obtained by taking a picture of a state in a conference room. The image pickup data may be in a format that can be displayed on the display device 10, and is, for example, still image data such as JPG, GIF, and PNG. Further, the frequency of shooting may be set to a time and the number of shots taken within that time, such as 1 shot per second or 12 shots per minute. The frequency of this shooting may be a preset interval or a user-set interval.

The screen data 130 captures the screen displayed in the conference and stores it as image data. For example, the conference material may be projected on the display unit 180 of the display device 10 and the display unit 180 may be captured periodically. The image data may be in a format that can be processed by the display device 10, and may be, for example, still image data such as JPG, GIF, or PNG. Further, the frequency of shooting may be set to a time and the number of shots taken within that time, such as 1 shot per second or 12 shots per minute. The frequency of this shooting may be a preset interval or a user-set interval.

The evaluation information 132 will be described with reference to FIG. The evaluation information 132 stores the evaluation input by the conference participants during the conference. In the evaluation information 132, the conference name (for example, "20170403_regular meeting 1"), the evaluation date and time indicating the input date and time (for example, "2017/4/3 10:04:23"), and the input evaluation content An emotional tag (for example, "positive") representing the above is stored.

The evaluation contents input by the conference participants are input to the display device 10 by, for example, operating an application running on the mobile terminal device 20 described later. In addition, the emotion tag may be other than positive and negative, and for example, an evaluation such as "surprise" or "questioned" may be used.

The bookmark information 134 will be described with reference to FIG. The bookmark information 134 stores the time point of the bookmark entered by the conference participant, and for example, when looking back at the conference, it stores the time point considered to be the time point when it is necessary to look back. The bookmark information 134 stores the conference name (for example, "20170403_regular conference 1") and the bookmark date and time (for example, "2017/4/3 10:04:34").

The bookmark information is input to the display device 10 by, for example, operating an application running on the mobile terminal device 20 described later.

The importance table 136 will be described with reference to FIG. The importance table 136 stores the importance of each interval (for example, "1 minute") having a predetermined time length. In the importance table 136, the meeting name (for example, "20170403_regular meeting 1"), the section number for identifying the section (for example, "1"), and the section start date and time (for example, "2017/4/3 10:00:00") are displayed. ), The date and time of the end of the section (for example, "2017/4/3 10:01: 00"), and the importance of the section (for example, "1") are stored.

Here, the importance is a numerical expression of how important each section in a conference is in the conference. In the present embodiment, the importance of the most important section is set to "4", and the importance of the least important section is set to "1", which are expressed in four stages. In the example of FIG. 6, the section 54 having an importance of “4” is an important section, and the section 5 having an importance of “3” follows. These sections show that they are important. On the other hand, since the importance of sections 1 to 4, section 6, section 53, and section 55 is "1", it indicates that these sections are not important.

In the present embodiment, the importance is expressed as a numerical value, but the importance may be finely set in proportion so that the maximum is expressed as 100%, for example. In addition to the numerical value, it may be expressed by a class such as "high", "medium", or "low", or it may be expressed by a flag such as whether it is important or not.

The digest data 138 is data obtained by deleting data of unimportant scenes from the conference data 122 based on the importance. Similar to the conference data 122, the digest data 138 stores the basic information of the conference, the voice data, the imaging data, and the screen data. Here, a part or all of the image pickup data 128 and the screen data 130 of the conference data 122 may not be included in the image pickup data or the screen data of the digest data 138.

The audio input / output unit 140 is a functional unit capable of inputting / outputting audio. The input voice is processed and output as voice data to each function unit, or the voice is output based on the voice data. Specifically, it may be configured by a microphone, a speaker, or headphones, or a microphone or speaker of another terminal device may be used.

The video input unit 150 is a functional unit that inputs an external video and generates video data. For example, it is composed of a camera and the like.

The communication unit 160 is a functional unit for communicating with the mobile terminal device 20 and the button-type beacon device 30. For example, it may be a functional unit that can communicate by a wired LAN, a functional unit that can communicate by a wireless LAN, or a functional unit that performs short-range wireless communication such as Bluetooth (registered trademark). The LAN communication method may be Ethernet (registered trademark) or IEEE802.11a / b / g / n, or may be an LTE (registered trademark) communication network or a communication network such as WiMAX (registered trademark). May be good.

The operation unit 170 is a functional unit for giving an instruction to the display device 10 and inputting data. For example, it consists of a keyboard and a mouse.

The display unit 180 is a functional unit for displaying screens necessary for the operation of the user of the display device 10 and conference data. For example, it is composed of an LCD (liquid crystal display), an organic EL display, a projector, and the like. Further, it may be a touch panel integrally formed with the operation unit 170.

[1.3 Processing flow]
Next, the flow of processing in this embodiment will be described.

[1.3.1 Main processing]
FIG. 7 is a flowchart illustrating the main processing of the display device 10. First, the user uses the operation unit 170 to determine whether or not any operation has been performed (step S1002). The user's operation may be an input operation using a keyboard, or may be a selection operation by displaying a menu on the display unit 180 and operating a mouse.

When any operation is performed, it is determined whether the operation performed in step S1002 is a conference data storage operation (step S1002; Yes → step S1004). In the case of the conference data storage operation, the user is made to input the basic information of the conference, and the storage of the conference data is started (step S1004; Yes → step S1006).

For example, input the conference name as the basic information of the conference. In this case, the conference participant may input the conference name using the operation unit 170, or may have the display device 10 select a fixed phrase stored in advance. As a fixed phrase, "regular meeting", "emergency meeting", etc. can be considered. Further, in order to identify another conference having the same name, the conference name may be added with the date and time and the serial number. For example, "regular meeting 0001" may be used as the meeting name when "regular meeting" is selected as the fixed phrase.

Further, the start of storage of the conference data is specifically a process of storing the audio data 126 based on the audio input from the audio input / output unit 140, and the captured image data 128 and the captured image data of the conference screen at predetermined intervals. It means to start the process of storing the screen data 130. In addition, the one including the image pickup data and the screen data to be stored in this embodiment, or one of them is referred to as a conference image data.

Here, for the process of storing the conference image data at predetermined intervals, for example, a timer that periodically increases the counter is prepared, and when the predetermined count number is reached, the conference image data is stored and the counter is reset. You may. In this case, if the cycle in which the timer operates is 1 second and the predetermined count number is 15, one conference image data is stored every 15 seconds.

While the conference data is stored, it is determined whether or not the emotion tag has been received (step S1008). When the emotion tag input by the conference participant is received, the conference name, the date and time of reception, and the received emotion tag are stored as evaluation information 132 (step S1008; Yes → step S1010).

Further, it is determined whether or not the bookmark information has been received (step S1012). When the bookmark information is received, the conference name and the received date and time are stored as bookmark information 134 (step S1012; Yes → step S1014). In the present embodiment, the bookmark information transmitted from the mobile terminal device 20 to the display device 10 may include information that the bookmark button has been pressed.

In the present embodiment, the fact that the emotion tag is received and the bookmark information is received means that the emotion tag and bookmark information transmitted by operating the mobile terminal device 20 are displayed via the network in the communication unit 160 of the display device 10. Judgment is made based on whether it was received by. The emotion tag and bookmark information transmitted from the mobile terminal device 20 to the display device 10 are referred to as a participant evaluation signal (evaluation signal).

A method of transmitting a participant evaluation signal from the mobile terminal device 20 will be described with reference to FIG. First, the conference participants launch an application for transmitting a participant evaluation signal (step S1052). An operation example of the application will be described later.

Subsequently, it is determined whether or not there has been a button selection operation arranged on the display screen of the application (step S1054). If there is a button selection operation (step S1054; Yes), it is subsequently determined which button has been selected.

First, it is determined whether the button for transmitting the positive emotion tag is selected (step S1056). When the button is selected, a positive emotion tag is transmitted from the mobile terminal device 20 to the display device 10 (step S1056; Yes → step S1058). After transmission, the process returns to step S1054.

Subsequently, it is determined whether or not the button for transmitting the negative emotion tag is selected (step S1060). When the button is selected, a negative emotion tag is transmitted from the mobile terminal device 20 to the display device 10 (step S1060; Yes → step S1062). After transmission, the process returns to step S1054.

Subsequently, it is determined whether or not the button for transmitting the bookmark information has been selected (step S1064). When the button is selected, the bookmark information is transmitted from the mobile terminal device 20 to the display device 10 (step S1064; Yes → step S1066). After transmission, the process returns to step S1054.

In addition to the emotion tag or bookmark information, the participant evaluation signal may include information on the transmission date and time. When the transmission date and time is included in the participant evaluation signal, the date and time included in the participant evaluation signal can be stored as the evaluation date and time of the evaluation information 132 or the bookmark date and time of the bookmark information 134 instead of the reception date and time of the display device 10. good.

Subsequently, it is determined whether or not the application has been terminated (step S1068). If there is an operation to terminate the application (step S1068; Yes), the application for transmitting the participant evaluation signal transmission is terminated. The termination operation may be an operation of terminating the application or an operation of turning off the power of the mobile terminal device 20.

An operation example of the application for transmitting the participant evaluation signal will be described with reference to FIG. On the application display screen W100, a Like button (like button) E100 for transmitting a positive emotion tag, an Hmm button (um button) E102 for transmitting a negative emotion tag, and bookmark information are displayed. A Bookmark button E104 for transmitting is displayed. The conference participant can touch the Like button, the Hmm button, and the Bookmark button at any time while the conference data is being recorded. When these buttons are touched by the conference participants, the mobile terminal device 20 transmits a participant evaluation signal to the display device 10 in response to the touched buttons.

The participant evaluation signal may be transmitted by operating the button-type beacon device 30, or the participant evaluation signal to be transmitted may be changed depending on how the button is pressed. For example, in the case of a button-type beacon terminal that can determine single-click, double-click, and long-press, or a button-type input terminal, a single-click sends a positive emotion tag, and a double-click sends a negative emotion tag. You may send and press and hold to send bookmark information.

Returning to FIG. 7, it is subsequently determined that the conference has ended (step S1016). The end of the conference is determined by, for example, displaying a button for instructing the end of the conference on the display unit 180 of the display device 10 and performing the button operation input from the operation unit 170 by the conference participants. You may. Further, it may be determined that the conference has ended when a predetermined time has elapsed. If the meeting has not ended, the process returns to step S1008 (step S1012; No).

When the conference is completed (step S1016; Yes), the storage of the conference data is stopped (step S1018). That is, the storage of the audio input from the audio input / output unit is stopped, and the process of storing the conference image data at predetermined intervals is stopped.

Subsequently, the importance determination process is performed (step S1020), and the optimization process is performed (step S1022). Details of the importance determination process and the optimization process will be described later.

Subsequently, it is determined whether the user's operation performed in step S1002 is a digest data reproduction operation (step S1024). If it is a digest data reproduction operation, the digest data reproduction process described later is performed (step S1024; Yes → step S1026).

Subsequently, it is determined whether the user's operation performed in step S1002 is an end operation (step S1028). The end operation may be an operation of turning off the power or an operation of terminating the program. If it is an end operation, the main process is terminated (step S1028; Yes). If it is not an end operation, the process returns to step S1002 (step S1028; No).

[1.3.2 Importance determination process]
Subsequently, the importance determination process will be described with reference to FIG. The importance determination process is a process of calculating the importance of the meeting for each section based on the emotion tag stored in the evaluation information 132 after stopping the storage of the meeting data.

First, for the conference that is the target of the importance determination process, the feature amount based on the emotion tag in the section is determined for each section (step S1102). The feature amount is a numerical value representing a specific characteristic when attention is paid to a specific characteristic in data related to the conference data such as evaluation information, voice data, and conference image data. In the present embodiment, the feature amount is calculated for each section from the evaluation date and time and the emotion tag stored in the evaluation information 132 by using the emotion tag included in the section.

The following can be adopted as a method (determination method) for calculating the feature amount based on the emotion tag.

(Method 1)
This is a method in which the ratio of which emotion tag is concentrated in which section is used as the feature quantity. Specifically, it is calculated by the following procedure. In addition, this feature amount is referred to as negative / positive degree.
(1) The overall mean value of positives is defined as a value obtained by dividing the total number of positive emotion tags received during the conference of the conference to be processed by the number of sections of the conference.
(2) The overall mean value of negatives is defined as a value obtained by dividing the total number of negative emotion tags received during the conference of the conference to be processed by the number of sections of the conference.
(3) Obtain the section value for each section. Here, the interval value is the value obtained by multiplying the number of positive emotion tags included in the section of interest by the positive overall average value, and the number of negative emotion tags included in the interval and the negative overall average value. It is defined as the value obtained by multiplying and subtracting. However, if it is less than 0, the section value of that section is set to 0.
(4) Of the interval values obtained in (3), the maximum one is defined as the maximum interval value.
(5) The negative / positive degree of the section of interest is calculated by multiplying the section value of the section by the maximum section value by 100.

(Method 2)
This is a method of using the ratio of the input of emotion tags concentrated in which section as a feature quantity. Physically, it is calculated by the following procedure.
(1) As the overall average value, it is defined as a value obtained by dividing the total number of emotion tags transmitted during the conference of the conference to be processed by the number of sections of the conference.
(2) Obtain the section value for each section. Here, the interval value is defined as a value obtained by multiplying the number of emotion tags included in the section of interest by the overall average value.
(3) Of the interval values obtained in (2), the maximum one is defined as the maximum interval value.
(4) The feature amount of the section of interest is calculated by multiplying the section value of the section by the maximum section value by 100.

(Method 3)
This is a method in which the ratio of the concentration of positive emotion tag inputs in which section is used as the feature quantity. Calculate according to the following procedure.
(1) As the overall average value, it is defined as a value obtained by dividing the total number of positive emotion tags transmitted during the conference of the conference to be processed by the number of sections of the conference.
(2) Obtain the section value for each section. Here, the interval value is defined as a value obtained by multiplying the number of positive emotion tags included in the interval of interest by the overall average value.
(3) Of the interval values obtained in (2), the maximum one is defined as the maximum interval value.
(4) The feature amount of the section of interest is calculated by multiplying the section value of the section by the maximum section value by 100.

Of course, the feature amount may be determined by a method other than those listed above. For example, in the method 3, the value calculated by using the negative emotion tag may be used as the feature amount. Further, among the emotion tags included in the section, the number of positive emotion tags and the number of negative emotion tags may be used as feature quantities, and the number of positive emotion tags may be used to determine the number of negative emotion tags. The feature quantity may be obtained by subtracting the number.

Subsequently, the importance is determined from the feature amount for each section (step S1104). As a method of determination, for example, it may be determined depending on which numerical range the feature amount calculated in step S1102 is included in. That is, the importance is determined such that "4" is for the section with the highest feature amount and "3" is for the section with the next highest feature amount. The numerical range may be predetermined or may be determined by the user. Further, the setting of the numerical range may be stored in the storage unit 120.

Next, the conference name of the conference whose importance is to be determined, the section number, the start date and time of the section, the end date and time of the section, and the importance of the section determined in step S1104 are stored in the importance table 136. Then (step S1106), the importance determination process ends.

An operation example of the importance determination process will be described with reference to FIG. FIG. 11A shows the section number, the section start date / time, the section end date / time, the number of positive emotion tags, the number of negative emotion tags, the negative / positive degree, and the importance. .. Further, FIG. 11B is a correspondence table between the negative / positive degree and the importance. In this example, the importance is determined by which numerical range in the table of FIG. 11B the negative / positive degree is included.

The negative / positive degree in FIG. 11A is a feature amount obtained by the method described in Method 1 among the above-mentioned feature amount calculation methods. In the example of FIG. 11A, the overall mean value of positives is 10 obtained by subtracting 5 which is the number of sections from 50 which is the number of positives of the entire conference. Further, the negative overall mean value is "3", and the maximum interval value is 194 of the interval in which the interval number is "3". Further, for example, the negative / positive degree of the section whose section number is "1" is calculated as about 25.77 by subtracting 194 from the section value of the section and multiplying by 100.

Next, the importance is determined for each section from the negative / positive degree and the table shown in FIG. 11 (b). For example, the negative / positive degree of the section where the section number is "2" is 72.68 ..., and since this value is included in the range where the negative / positive degree is 60 or more and less than 75, the importance of the section is determined to be "3". do.

[1.3.3 Optimization processing]
Subsequently, the optimization process will be described with reference to FIG. The optimization process is a process of creating and storing digest data 138 from the conference data 122 by deleting the conference image data according to the importance of the section.

First, the importance of the head section of the conference to be optimized is read from the importance table 136 (step S1152).

Subsequently, among the conference image data in the section, the conference image data to be deleted is determined according to the read importance (step S1154). That is, which of the conference image data included in the section of interest is to be deleted from the conference data to be optimized, is determined according to the importance of the section of interest. do.

The conference image data to be deleted and the ratio of the conference image data to be deleted are determined by any appropriate method. For example, the ratio of conference image data to be deleted is 0% if the importance is "4", 50% if the importance is "3", 75% if the importance is "2", 100% if the importance is "1", and so on. It should be decided as follows.

Since the above processing is performed in all sections, it is determined whether or not the importance of all sections of the target conference has been read (step S1156). If not all have been read, the importance of the next section is read (step S1156; No → step S1158), and the process returns to step S1154.

An operation example of the optimization process will be described with reference to FIG. FIG. 13 shows a graph in which the importance is on the vertical axis and the elapsed time is on the horizontal axis, and the conference image data G100 stored corresponding to the elapsed time is arranged side by side. Further, the conference image data after the optimization process is shown as the conference image data G102 corresponding to the elapsed time.

In the example of FIG. 13, it is shown that four sections of t0 to t1, t1 to t2, t2 to t3, and t3 to t4 are included as sections. Further, it is shown that four conference image data are stored for each section.

Here, the ratio of the conference image data to be deleted is 0% if the importance is "4", 50% if the importance is "3", 75% if the importance is "2", and 100% if the importance is "1". And. At this time, since the interval from t0 to t1 has an importance of 1, all the conference image data included in this interval is deleted. In the present embodiment, the conference image data to be optimized is the conference data at the start of the section, but may be the conference image data at the end of the section. Further, since the section from t1 to t2 has an importance of 4, the conference image data included in this section is not deleted. Since the section from t2 to t3 has an importance of 3, 50% of the conference image data included in this section is deleted. Here, the two conference image data at the time point of one quarter of the section and the time point of three quarters of the section are deleted. Since the section from t3 to t4 has an importance of 2, 75% of the conference image data included in this section is deleted.

As described above, in the conference image data G102, the conference image data at the time indicated by the net dots is to be deleted, and the conference image data at the time not indicated by the mesh dots is included in the digest data in the digest data storage process described later. It will be stored as conference image data.

In the present embodiment, the importance is stored in the importance table 136 in step S1106, and the importance is read out again in step S1152. However, once the importance is determined in step S1104, the importance is subsequently determined. The conference image data may be deleted according to the situation.

Returning to FIG. 12, when the importance of all the sections is read out, the digest data is subsequently stored (step S1156; Yes → step S1160). That is, the basic information 124 of the conference data 122, the audio data 126, the image pickup data 128 that was not deleted in step S1154, and the screen data 130 are stored as digest data 138.

As a result, the digest data 138, which has a smaller amount of data than the conference data 122, is stored. After storing the digest data 138, the original conference data 122 may be deleted. By leaving only the digest data 138, the amount of data stored by the display device 10 can be reduced.

[1.3.4 Digest data playback processing]
Subsequently, the digest data reproduction process will be described with reference to FIG.

First, the digest data 138 stored in the storage unit 120 is displayed in a list (step S1202). Regarding the list display, for example, the conference name of the basic information included in the digest data 138 may be displayed as a list, or other basic information or a part of the conference image data may be displayed.

Subsequently, when the digest data 138 is selected by the user (step S1204; Yes), a negative / positive graph based on the number of positive emotion tags and the number of negative emotion tags is displayed (step S1206), and the negative / positive graph is displayed. The bookmark information is displayed in (step S1208).

The negative / positive graph may display the negative / positive degree obtained in step S1102, or may display the importance stored in the importance table 136. Further, the change in the feature amount in the entire section of the conference may be displayed, the change in the feature amount before and after the reproduction time may be enlarged and displayed, or both may be displayed. That is, changes in the feature amount and the importance may be appropriately displayed in correspondence with the time of reproduction.

Further, the bookmark information may be displayed on the graph at the corresponding position of the negative / positive graph displayed in step S1206, or may be displayed as a list in an area different from the negative / positive graph.

Subsequently, the reproduction of the audio data included in the digest data 138 selected in step S1204 is started (step S1210). In addition, the display of the negative / positive graph is updated according to the playback time point (step S1212). For example, when the display indicating the reproduction position is displayed on the negative / positive graph, the display indicating the reproduction position is updated so as to have an appropriate position and size. Further, when the negative / positive graph is displayed so as to be enlarged before and after the reproduction time point, the enlargement range is changed according to the change of the reproduction time point.

Subsequently, if the conference image data at the time of reproduction is stored, the conference image data is displayed (step S1214; Yes → step S1216). Regarding the display of the conference image data, for example, the time of the actual conference corresponding to the playback time is obtained from the start date and time of the conference and the playback time of the voice, and the latest conference image data stored before the time is obtained. Should be displayed.

Subsequently, it is determined whether the user has selected the seek bar or the negative / positive graph displayed in step S1206 (step S1218). When the seek bar or the negative / positive graph is selected, the reproduction time point is moved to the selected position (step S1218; Yes → step S1220). After that, by returning to step S1212, the negative / positive graph and the conference image data at the time of movement are displayed.

Subsequently, it is determined whether or not the operation for ending the digest data reproduction process has been performed (step S1222). When the end operation is performed, the digest display process is terminated (step S1222; Yes). If the end operation is not performed (step S1222; No), the process returns to step S1212 and the reproduction of the digest data 138 is continued.

An operation example of the digest data reproduction process will be described with reference to FIG. The display screen W110 displays a list E110 for selecting digest data 138, a label E112 for displaying the actual meeting time at the time of reproduction, a negative / positive graph E114 based on the negative / positive degree, a seek bar E118, and screen data 130. The area E120 for displaying the image data 128, the area E122 for displaying the image pickup data 128, the area E124 for displaying the basic information of the conference, the button E126 for instructing the sound reproduction, and the button E128 for instructing to stop the sound reproduction are displayed. ing. Further, the bookmark information is displayed superimposed on the negative / positive graph E114 as shown in E116.

In this example, the start date and time of the conference is "10:00:00 on April 3, 2017", the length of the conference is "54:30", and the playback time is "10:06". It is at the time of "minute 45 seconds". Therefore, "10:06:45" is displayed on the label E112 that displays the time of the actual meeting.

Further, the length of one section is 1 minute, and in the negative / positive graph E114, the vertical axis is the number of positive emotion tags and the number of negative emotion tags in each section on two axes, and the horizontal axis is the actual number. The time of the meeting is displayed. For example, the point struck at the time of "11:00:00" is a positive input during one minute from "10:00:00" to "10:11:00". It is the sum of the number of emotion tags or the number of negative emotion tags.

By doing so, the user can play the digest data while checking the part where the conference participants responded, while the conference image data is deleted based on the importance, so the digest The amount of data can be reduced.

[2. Second Embodiment]
Next, the second embodiment will be described. The second embodiment is an embodiment in which, in the digest data reproduction process, not the entire digest data but only a portion having high importance is reproduced. The second embodiment has the same functional configuration as the first embodiment. Further, since the processing is almost the same, the description thereof will be omitted, and the contents different from those of the first embodiment will be mainly described.

This embodiment is an embodiment in which the process of FIG. 14 of the first embodiment is replaced with the process of FIG. After selecting the digest data 138 in step S1204, the reproduction section narrowing process is performed (step S2002). The reproduction section narrowing process will be described later. If the user's operation is a reproduction section narrowing operation, the process returns to step S2002 (step S2004; Yes).

Here, the reproduction section refers to a section to be reproduced in the present embodiment, and the reproduction section narrowing is to extract a section of the digest data 138 that meets the conditions specified by the user and reproduce the section. It means to make it a section. As the narrowing-down method in the present embodiment, the user can specify either a method of designating the importance or a method of designating the reproduction time.

The reproduction section narrowing process will be described with reference to FIG. First, it is determined whether the narrowing method is a method for designating the importance (step S2052).

If the narrowing method is a method of designating the importance, the user is made to select the threshold of the importance to be extracted (step S2052; Yes → step S2054). Subsequently, when the threshold value of importance is specified, it is extracted as a section to be reproduced based on the threshold value of the designated importance (step S2056). For example, the section of the digest data 138 whose importance is equal to or higher than the specified importance may be extracted, or the section of the digest data 138 whose importance is less than the specified importance may be extracted by excluding it from the reproduction target. good.

Subsequently, the extracted section is set as the section to be reproduced (step S2070), and the reproduction section narrowing process is completed.

When the narrowing-down method is the specification of the reproduction time, the user is made to specify the reproduction time (step S2052; No → step S2058). Then, the section to be reproduced is extracted based on the specified reproduction time.

In the extraction of the section to be reproduced based on the reproduction time, first, the section having the highest importance is extracted (step S2060). Subsequently, it is determined whether the length of the extracted section matches the reproduction time specified in step S2058 (step S2062). If they match, the extracted section is set as the section to be reproduced, and the reproduction section narrowing process is terminated (step S2062; Yes → step S2070).

If the length of the extracted section does not match the reproduction time specified in step S2058, it is determined whether the length of the extracted section exceeds the reproduction time (step S2064). If the reproduction time is not exceeded (step S2064; No), a section having a lower importance by one step is extracted (step S2068), and it is determined whether or not the reproduction time is within the reproduction time (step S2062).

If the reproduction time is exceeded (step S2064; Yes), the extracted sections are narrowed down so as to fit within the reproduction time (step S2066). As an example of narrowing down the extracted sections so as to fit in the playback time in this case, the sections up to the importance within the playback time may be extracted as the sections to be played back, or the sections may be narrowed down to fit in the playback time based on the feature amount. In addition, it may be preferentially included in the section to be reproduced in order from the size of the feature amount.

An operation example of narrowing down the reproduction section will be described with reference to FIG. FIG. 18A is a two-axis graph showing the importance when the importance is determined using only the positive emotion tag and the importance when the importance is determined using only the negative emotion tag (a). For example, the upper side is represented by a positive emotion tag, and the lower side is represented by a negative emotion tag). For example, the points struck at the time of "11:00:00" are the positive emotion tag and the negative emotion tag in the section from "10:00:00" to "10:11:00". It shows that each importance calculated by using the above is 1.

Further, it indicates that the bookmark information b1 is stored between t1 and t2, the bookmark information b2 is stored between t4 and t5, and the bookmark information b3 is stored between t7 and t8.

For example, when it is decided to reproduce only a section having an importance of 2 or more, the section surrounded by a rectangle in FIG. 18A is a section having an importance of less than "2", that is, a section having an importance of "1". This is the part where a certain section continues. This part is not included in the reproduction section because it is not necessary to look back when looking back on the digest data 138.

As a result, as shown in FIG. 18B, from the digest data 138, between the sections t1 and t2, between the sections t3 and t4, between the sections t5 and t6, and between the sections t7 and t8. The section between t9 and t10 is extracted as the section to be reproduced. Since the section t4 and t5 in which the bookmark information b2 exists is not included in the reproduction section, the bookmark information may not be displayed when the digest data 138 is reproduced.

In addition, in this embodiment, the section of importance "1" in the adjacent section where the importance is "2" or more is included in the reproduction section. For example, "10:04: 00" is included as a reproduction target section for the time before and after the importance of the positive emotion tag / negative emotion tag other than "1".

An operation example of the digest data reproduction processing in the present embodiment will be described with reference to FIG. On the display screen W200, in addition to each element displayed on the display screen W110, an area E200 for setting a narrowing down of a section to be reproduced is displayed. Further, instead of the graph E114, the graph E202 is displayed by extending the section to be reproduced to the entire screen and adding the display of the boundary line indicating the boundary with the continuous section.

In addition, the graph E202 may display the feature amount and the importance in another appropriate format such as enlarging the section corresponding to before and after the reproduction time. Further, instead of the display screen W200, the display screen W210 shown in FIG. 20 may be displayed. On the display screen W210, an area E210 for displaying the conference image data, a button E212 for switching playback and stop, a seek bar E214, and a label E216 for displaying the actual conference time at the time of reproduction are displayed.

Here, as the conference image data, both the captured image data and the screen data may be displayed, or either one may be displayed. Further, the seek bar may display a boundary line indicating a boundary with a continuous section.

By doing so, the user can efficiently look back on the digest data 138 by narrowing down the section of the digest data 138 having a high importance and reproducing only the section. Further, since the section that can be reproduced is limited, the reproduction time point that can be selected by the user is also limited, so that improvement in operability can be expected. Further, by limiting the elements to be displayed on the display screen such as the display screen W210, it is possible to look back on the conference while maintaining convenience even in a display device having a low resolution.

Further, by applying the present embodiment, the digest data after narrowing down the reproduction section may be collectively downloaded as one moving image file. By doing so, the digest data 138 can be reproduced if the terminal can reproduce the moving image. In this case, in order to make the playback time of the video file shorter than the actual conference, the audio data of the section not to be reproduced is deleted from the audio data 126, and the video is created from the shortened audio data and the corresponding conference image data. It should be configured.

[3. Third Embodiment]
Next, the third embodiment will be described. The third embodiment has the same configuration as the first embodiment, but is an embodiment in which two feature quantities, a feature amount obtained from the emotion tag and an average volume value, are used for determining the importance. Since the third embodiment has the same functional configuration as the first embodiment and the processing is almost the same, the description thereof will be omitted and the contents different from those of the first embodiment will be mainly described. Further, the feature amount calculated in step S1102 will be described as a negative / positive degree.

This embodiment is an embodiment in which the process of FIG. 10 of the first embodiment is replaced with the process of FIG. 21. First, for each section of the conference that is the target of the importance determination process, the negative / positive degree is calculated as the first feature amount based on the emotion tag in the section (step S1102). Subsequently, the importance determination value is determined from the negative / positive degree (step S3002). Here, the importance determination value is a value expressed as a uniform value by absorbing a range of values that each feature amount can take when dealing with a plurality of different feature amounts.

Subsequently, the average volume value for each section is calculated as the second feature amount (step S3004). As the average volume value, the average value of the volume of the section of interest of the voice data 126 may be used. Subsequently, the importance determination value is determined from the average volume value (step S3006).

Subsequently, the importance of each section is determined from the importance determination values calculated in steps S3002 and S3006, and stored in the importance table 136 (step S3008). As a method of determining the importance, for example, the importance may be determined based on the value obtained by adding the importance determination value. In this case, each importance determination value may be weighted. Further, the importance may be determined based on the larger of the importance determination value of the negative / positive degree and the importance determination value of the average volume value.

An operation example of the importance determination process in the present embodiment will be described with reference to FIGS. 22 and 23. FIG. 22A is a table showing the section number, the section start date and time, the section end date and time, the negative positive degree which is the first feature amount, and the average volume value which is the second feature amount. Further, FIG. 22 (b) shows the correspondence between the importance judgment value and the first and second feature quantities, and FIG. 22 (c) shows the correspondence between the total value of the importance judgment value and the importance. Shown.

FIG. 22D is a table showing the first and second feature quantities for each section, the respective importance determination values, the total value of the respective importance determination values, and the importance. For example, in the section having the section number “2”, the negative / positive degree, which is the first feature amount, is “80”, so the importance determination value of the negative / positive degree is set to “5” from the table of FIG. 22 (b). Further, since the average volume value, which is the second feature amount, is "75 dB", the importance determination value of the average volume value is set to "5" from the table of FIG. 22 (b). Further, since the sum of the importance determination values is "10", the importance of the section is determined to be "4" from FIG. 22 (c).

FIG. 23 graphically represents the negative positive degree, which is the first feature amount shown in FIG. 22 (a), and the average volume, which is the second feature amount shown in FIG. 22 (b), and performs the optimization process. It is a figure which showed the meeting image data G104 of the state after having done. For example, in the section from t1 to t2, since the volume and the negative / positive degree are high, the importance is set to "4" and all the conference image data is stored. On the other hand, in the section from t3 to t4, the negative / positive degree is low, and the volume is about half that in the section from t1 to t2. Therefore, the importance is determined to be "2", and one of the four conference image data included in the section from t3 to t4 is stored.

By calculating the importance from a plurality of features in this way, it is possible to comprehensively determine important sections in a meeting. For example, in a conference, it is assumed that the conference participants concentrate on the conversation and forget to operate the mobile terminal device 20. Even in that case, if the feature amount based on the average volume value is large, it can be expected that the section is judged to be important.

Further, FIG. 24 shows an operation example of the digest data reproduction process in the present embodiment. The processing content of the digest data reproduction processing is substantially the same as that of the digest data reproduction processing of the first embodiment, but in step S1206, it was determined from the negative / positive degree and the average volume value, which are the feature quantities, instead of the negative / positive graph. A graph based on the importance determination value may be displayed. For example, the importance determination value based on the negative / positive degree is displayed in the upper half of the graph, and the importance determination value based on the average volume value is displayed in the lower half of the graph. In addition, each feature amount may be displayed.

By displaying features based on negative positives and average volume values, the reason why the section of interest is important is due to the evaluation of the conference participants, the volume, or both. Can be determined. By doing so, the user can look back on important scenes while checking the changes in each feature amount.

In the present embodiment, the importance is determined by using the negative / positive degree and the average volume value, but it goes without saying that other feature quantities may be used. For example, the amount of change in the conference image data may be used as the feature amount. This is important because the scene with a large amount of change in the image is actively moving and the situation of the place is changing. Further, as the feature amount, the average of the change amounts of the pixel values per pixel may be used for the conference image data in the section.

[4. Fourth Embodiment]
Next, the fourth embodiment will be described. The fourth embodiment has the same configuration as the second embodiment, but is an embodiment in which the reproduction section is extracted including the front and back of the section having high importance. The fourth embodiment has the same functional configuration as the second embodiment. Further, since the processing is almost the same as that of the second embodiment, the description thereof will be omitted, and the contents different from those of the second embodiment will be mainly described.

In the present embodiment, in steps S2056 and S2060, a section in which three sections are added before the high-importance section and one section is added after the high-importance section is extracted as the high-importance section. As a result, it is possible to reproduce including the context of the section of high importance.

An operation example of this embodiment will be described with reference to FIG. 25. The configuration of the display screen W400 is the same as that of W200 in FIG. 19, but the reproduction section is a section including a section having high importance and a section before and after the section, and in the graph E400, the extracted sections are combined. Is displayed.

The length of the section to be included before and after the section of high importance may be determined in advance, for example, "3 sections before, 1 section after" in the present embodiment, and the feature amount of the audio data or the conference image data. It may be dynamically determined until the change of is equal to or less than a predetermined threshold value.

By doing so, it is possible to look back on the context including the context that leads to the section of high importance, so it is expected that the effect of being able to look back on the meeting more clearly can be expected.

[5. Fifth Embodiment]
Next, the fifth embodiment will be described. The fifth embodiment has the same configuration as the first embodiment, but is an embodiment that analyzes voice data and determines the importance in consideration of the number of speakers and conference environment information such as speakers. ..

For example, in the section where there is only one speaker, the volume is low, so the feature amount of the average volume value is small. Here, if a section with one speaker continues, the average volume value may be corrected by regarding the section as an important section, or a feature amount related to the conference environment information may be provided to determine the importance. You may use it.

By doing so, the importance can be determined according to the conference environment information. For example, a section in which one person is speaking can be considered as a section in which an announcement is being made, and can be regarded as an important section. In this case, it can be expected that the section in which the announcement is made and the section in which the reaction is made can be extracted by performing the reproduction section narrowing process.

[6. 6th Embodiment]
Next, the sixth embodiment will be described. The sixth embodiment is an embodiment in which the importance is determined in consideration of biological information. Examples of biometric information include brain waves, heart rate, respiratory rate, and the like of conference participants.

The sixth embodiment is an embodiment in which the overall configuration of FIG. 1 of the first embodiment is replaced with the overall configuration of FIG. 26. As shown in FIG. 26, in the conference support system 1, a plurality of mobile terminal devices 20 and button-type beacon devices 30 are connected to the display device 10. Further, a plurality of biometric information measuring devices 40 are connected to the display device 10. In FIG. 26, the biometric information measuring device 40 is displayed as being connected to the display device 10 by the same network, but may be connected via another network. Further, it may be connected by wire.

It is assumed that the biological information measuring device 40 is a device capable of measuring the biological information of the person who wears the device. For example, it may be a smart watch or a headset-type electroencephalograph.

The measurement result of the biometric information may be transmitted to the display device 10 at any time during the storage of the conference data, the biometric information is stored during the conference, and the data is transmitted to the display device 10 after the conference is completed. You may.

In order to determine the importance from the biometric information, the importance may be determined directly from the biometric information. Further, the importance determination value may be obtained from the biological information, and the importance may be determined from the importance determination value and another importance determination value.

By doing so, the importance is comprehensively determined using the features that take into account the biometric information of the meeting participants, in addition to the features that are based on active movements such as evaluations and remarks by the meeting participants. You can expect to be able to do it.

[7. Seventh Embodiment]
Next, the seventh embodiment will be described. The seventh embodiment is an embodiment in which the importance is determined based on the average volume value.

Specifically, for example, in the importance determination process of FIG. 10, instead of performing the process of calculating the feature amount based on the emotion tag for each section, the average volume value for each section is obtained and based on the average volume value. You just have to decide the importance. Alternatively, in the importance determination process of FIG. 21, the importance may be determined by setting the weight of the importance determination value determined from the feature amount of the emotion tag to 0.

By doing so, even when the mobile terminal device 20 or the button-type beacon device 30 cannot be used, the digest data can be created only from the conference data.

[8. Eighth Embodiment]
Next, the eighth embodiment will be described. The eighth embodiment is an embodiment based on a client-server system in which a device for storing conference data and a device for creating and storing digest data are separated.

The eighth embodiment is an embodiment in which the overall configuration of FIG. 1 of the first embodiment is replaced with the overall configuration of FIG. 27. Here, the display device 10 and the server device 50 are connected via a network. In FIG. 27, the display device 10 and the server device 50 are represented as existing in the same network, but each device may exist in a different network.

Regarding the main processing, the operation of the display device 10 and the server device 50 on the client side will be described with reference to FIG. 7. For example, in step S1006, when the basic information of the conference is input to the display device 10, the input information is stored in the display device 10 and transmitted to the server device 50. When the server device 50 receives the transmitted data, the server device 50 can detect that the conference has started.

Subsequently, the conference image data periodically stored by the display device 10 and the participant evaluation signal received by the display device 10 in steps S1008 and S1012 are transmitted to the server device 50 together with the conference name. The server device 50 stores the received conference image data and the participant evaluation signal in association with the conference name.

Further, in step S1018, when the display device 10 stops the storage of the conference data, the audio data is transmitted to the server device 50 together with the conference name. When the server device 50 stores the received voice data in association with the conference name, the server device 50 performs the importance determination process in step S1020 and the optimization process in step S1012.

Further, in step S1024, when the display device 10 reproduces the digest data, the display device 10 may acquire the digest data stored in the server device 50.

As described above, the server device 50 performs a part of the main processing, so that the display device 10 stores the conference data and sends it to the server device 50, receives the participant evaluation signal, and sends it to the server device 50. Will be done. This makes it possible to create and view digest data even if the display device 10 is a device having low performance.

Further, since the amount of conference image data transmitted from the server device 50 to the display device 10 is reduced by the optimization process as compared with the original conference data, the digest data can be browsed efficiently. ..

[9. Modification example]
Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to the embodiments, and the design and the like within a range not deviating from the gist of the present invention are also within the scope of claims. included.

Although the above-described embodiments are described separately for convenience of explanation, they can be combined and executed to the extent possible. For example, a button for switching between the display screen W110 for reproducing the entire conference and the display screen W200 may be displayed by combining the first embodiment and the second embodiment.

Although the above-described embodiment is described for a conference, it is needless to say that the embodiment may be applied to other than the conference. For example, it may be applied to an environment similar to a conference such as training or class, or may be applied to a field that handles both audio data and video data (for example, data recording sports).

Further, although it is explained that the above-described embodiment includes the imaging data and the screen data as the conference image data, it may be an embodiment in which only one of them is used, or an embodiment in which the conference image data is not used. There may be. Further, the types of imaging data may be increased by using a plurality of cameras, or a plurality of screens may be captured to increase the types of screen data. On the digest playback screen, as many areas as the number of types of conference image data stored as conference data may be provided to display the conference image data.

Further, the program that operates in each device in the embodiment is a program that controls a CPU or the like (a program that causes a computer to function) so as to realize the functions of the above-described embodiment. The information handled by these devices is temporarily stored in a temporary storage device (for example, RAM) at the time of processing, and then stored in a storage device such as an HDD or SSD (solid state drive), if necessary. Read, modify and write by the CPU.

Further, in the case of distribution to the market, the program can be stored and distributed in a portable recording medium, or transferred to a server computer connected via a network such as the Internet. In this case, it goes without saying that the storage device of the server computer is also included in the present invention.

10 Display device 110 Control unit 120 Storage unit 122 Conference data 124 Basic information 126 Audio data 128 Imaging data 130 Screen data 132 Evaluation information 134 Bookmark information 136 Importance table 138 Digest data 140 Audio input / output unit 150 Video input unit 160 Communication unit 170 Operation unit 180 Display unit 20 Portable terminal device 30 Button type beacon device

Claims (17)

A digest data generator that can be connected to a plurality of external devices that transmit information selected by the user among the first information indicating positive emotions and the second information indicating negative emotions as evaluation signals.
Image data acquisition means for acquiring image data and
A storage means for storing the audio data and the image data periodically acquired corresponding to the audio data as content data, and
A receiving means for receiving the evaluation signal and
A feature amount determining means for determining a feature amount based on the first information and the second information included in the evaluation signal for each interval of a predetermined time length with respect to the content data.
An importance determination means for determining the importance of each section from the feature amount for each section,
A digest data generation means for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
A digest data generator characterized by being equipped with. The feature amount determining means sets the total number of the evaluation signals including the first information to the number of the evaluation signals including the first information for the section of interest for the content data for each section. From the value obtained by multiplying the value divided by the number of the sections, the total number of the evaluation signals including the second information is added to the number of the evaluation signals including the second information for the section of interest. The value obtained by subtracting the value obtained by multiplying the data by the number of the sections is the section value of the section of interest, and the section value of the section is obtained for each section as the feature amount for the section. The digest data generation device according to claim 1, wherein the feature amount is a value obtained by multiplying the maximum value divided by the section value among the section values by 100. The feature amount determining means is characterized by a value obtained by subtracting the number of the evaluation signals including the second information for the section from the number of the evaluation signals containing the first information for the section as the feature amount for the section. The digest data generation device according to claim 1, wherein the amount is used. A digest data generator that can be connected to a plurality of external devices that transmit information selected by the user among the first information indicating positive emotions and the second information indicating negative emotions as evaluation signals.
Image data acquisition means for acquiring image data and
A storage means for storing the audio data and the image data periodically acquired corresponding to the audio data as content data, and
A receiving means for receiving the evaluation signal and
With respect to the content data, a feature amount for each section having a predetermined time length is determined for each feature based on a plurality of different features including the first information included in the evaluation signal and the feature based on the second information. Means for determining the amount of features to be used,
An importance determination means for determining the importance of each section based on the feature amount of each section determined for each feature, and
A digest data generation means for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
A digest data generator characterized by being equipped with. The feature amount determining means uses the feature amount based on the first information and the second information included in the evaluation signal as the first feature amount, and the feature amount based on the volume of the voice data as the second feature amount. The digest data generation device according to claim 4 , wherein the digest data is generated. Further equipped with an imaging means for capturing an image,
The digest data generation device according to any one of claims 1 to 5 , wherein the image data acquisition means acquires images captured in time series by the image pickup means as the image data. From claim 1, the digest data generation means generates the digest data from the content data by deleting the image data and the audio data according to the importance determined for each section. The digest data generation device according to any one of 6 . The digest data generating means is characterized in that moving image data is generated from the digest data from which the image data and the audio data are deleted based on the importance determined for each section from the content data. The digest data generator according to any one of 1 to 7 . In the digest data reproduction apparatus for reproducing the digest data generated in the digest data generation apparatus according to any one of claims 1 to 8 .
A reproduction time point selection means for selecting a reproduction time point of the audio data included in the digest data, and a reproduction time point selection means.
A graph display means for displaying a graph showing a time-series change of the feature amount corresponding to the reproduction time point, and a graph display means.
A digest data playback device characterized by being equipped with. The graph display means is characterized in that, for each section, a graph showing the number of the evaluation signals including the first information corresponding to the section and the number of the evaluation signals including the second information is displayed. The digest data reproduction device according to claim 9. The digest data according to claim 9 or 10, wherein the reproduction time point selection means limits selectable reproduction time points based on the importance of each section determined by the importance determination means. Playback device. A digest data generation system in which a server device , a display device, and a plurality of terminal devices are connected.
The terminal device is
A first transmission means for transmitting information selected by the user among the first information indicating positive emotions and the second information indicating negative emotions as an evaluation signal.
Equipped with
The display device is
Image data acquisition means for acquiring image data and
A content data transmission means for transmitting the voice data and the image data periodically acquired corresponding to the voice data to the server device as content data.
A second transmission means for transmitting the evaluation signal received from the terminal device to the server device, and
The server device is equipped with
A receiving means for receiving the content data and the evaluation signal from the display device, and
A feature amount determining means for determining a feature amount based on the first information and the second information included in the evaluation signal for each interval of a predetermined time length with respect to the content data.
An importance determination means for determining the importance of each section from the feature amount for each section,
A digest data generation means for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
A digest data transmitting means for transmitting the digest data to the terminal device,
A digest data generation system characterized by being equipped with. A digest data generation system in which a server device , a display device, and a plurality of terminal devices are connected.
The terminal device is
A first transmission means for transmitting information selected by the user among the first information indicating positive emotions and the second information indicating negative emotions as an evaluation signal.
Equipped with
The display device is
Image data acquisition means for acquiring image data and
A content data transmission means for transmitting the voice data and the image data periodically acquired corresponding to the voice data to the server device as content data.
A second transmission means for transmitting the evaluation signal received from the terminal device to the server device, and
The server device is equipped with
A receiving means for receiving the content data and the evaluation signal from the display device, and
With respect to the content data, a feature amount for each section having a predetermined time length is determined for each feature based on a plurality of different features including the first information included in the evaluation signal and the feature based on the second information. Means for determining the amount of features to be used,
An importance determination means for determining the importance of each section based on the feature amount of each section determined for each feature, and
A digest data generation means for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
A digest data transmitting means for transmitting the digest data to the terminal device,
A digest data generation system characterized by being equipped with. A method for generating digest data by a digest data generator that can be connected to a plurality of external devices that transmit information selected by the user among the first information indicating positive emotions and the second information indicating negative emotions as evaluation signals. And
Image data acquisition step to acquire image data and
A storage step of storing the audio data and the image data periodically acquired corresponding to the audio data as content data, and a storage step.
A reception step for receiving the evaluation signal and
A feature amount determination step of determining a feature amount based on the first information and the second information included in the evaluation signal for each interval of a predetermined time length with respect to the content data.
An importance determination step for determining the importance of each section from the feature amount for each section,
A digest data generation step for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
A digest data generation method comprising. A method for generating digest data by a digest data generator that can be connected to a plurality of external devices that transmit information selected by the user among the first information indicating positive emotions and the second information indicating negative emotions as evaluation signals. And
Image data acquisition step to acquire image data and
A storage step of storing the audio data and the image data periodically acquired corresponding to the audio data as content data, and a storage step.
A reception step for receiving the evaluation signal and
With respect to the content data, a feature amount for each section having a predetermined time length is determined for each feature based on a plurality of different features including the first information included in the evaluation signal and the feature based on the second information. The feature amount determination step to be performed and
An importance determination step for determining the importance of each section based on the feature amount of each section determined for each feature, and
A digest data generation step for generating digest data from which the image data is deleted according to the importance determined for each section from the content data.
A digest data generation method comprising.
To a computer that can be connected to a plurality of external devices that transmit information selected by the user among the first information indicating positive emotions and the second information indicating negative emotions as evaluation signals .
Image data acquisition function to acquire image data and
A storage function that stores audio data and the image data periodically acquired in response to the audio data as content data, and
The reception function for receiving the evaluation signal and
A feature amount determination function for determining a feature amount based on the first information and the second information included in the evaluation signal for each interval of a predetermined time length with respect to the content data.
An importance determination function for determining the importance of each section from the feature amount for each section, and
A digest data generation function that generates digest data from which the image data is deleted according to the importance determined for each section from the content data.
A program to execute. To a computer that can be connected to a plurality of external devices that transmit information selected by the user among the first information indicating positive emotions and the second information indicating negative emotions as evaluation signals .
Image data acquisition function to acquire image data and
A storage function that stores audio data and the image data periodically acquired in response to the audio data as content data, and
The reception function for receiving the evaluation signal and
With respect to the content data, a feature amount for each section having a predetermined time length is determined for each feature based on a plurality of different features including the first information included in the evaluation signal and the feature based on the second information. Feature amount determination function and
An importance determination function that determines the importance of each section based on the feature amount of each section determined for each feature, and
A digest data generation function that generates digest data from which the image data is deleted according to the importance determined for each section from the content data.
A program to execute.

Images