JP2019149083A - Minute creating apparatus, minute creating method, and program - Google Patents

Abstract

To provide an apparatus which creates a layout by associating data such as sound generated during a meeting, an image and the like with one another.SOLUTION: In a minute 1100 consisting of a plurality of minute page groups including text data and image data, a text obtained by speech recognition of audio data generated during a meeting and an image data are stored. A single meeting is divided into plural periods based on time information, and a layout 1120 (1121) is created by associating the sound data and the image data both of which are generated in the same time period.SELECTED DRAWING: Figure 11

Description

  The present invention relates to an apparatus for displaying conference information. In particular, the present invention relates to an apparatus that performs layout by associating data such as audio and images generated during a meeting.

  Conventionally, there is a dictation technique for recognizing a person's utterance and converting it into text. Patent Document 1 discloses a system for generating minutes by recognizing a person's utterance, converting it into text, and summarizing it. Patent Document 1 discloses a system that generates minutes by recognizing characters included in a video, converting them into text, and summarizing them.

Japanese Patent No. 5104762

A conference generally has a plurality of agendas, and discussions are held for each agenda. In addition, during the discussion, the conference may proceed while writing the content of the discussion on a whiteboard or the like. When the meeting is completed, the minutes of the meeting are often created.
At this time, to make it easier for users to look back on the discussion, for example, generate a minutes that is laid out by associating images written when discussing an agenda with text summarizing the content of the discussion on that agenda It is desirable if possible.

The minutes generated in Patent Document 1 are laid out with summarized text.
However, Patent Document 1 does not disclose, for example, the generation of the minutes that are summarized and laid out for each agenda. Also, there is no disclosure of generating a minutes by associating a text summarizing an utterance with an image written at that time.

  The present invention provides a first acquisition means for acquiring a text generated from audio data in conference information generated in a conference, and a first time at which the audio data is generated, and in the conference information Second acquisition means for acquiring image data and a second time when the image data occurred, first recording means for recording the text and the first time, and the image data Second recording means for recording the second time, dividing means for dividing the conference into a plurality of sections, and text generated from the start time to the end time of the sections, The first selection means to select from the text recorded in the recording means, and the image data generated from the start time to the end time of the section are recorded in the second recording means. Select from the data And a creation means for creating a layout by associating the text selected by the first selection means with the image data selected by the second selection means. Is a minutes generation device.

  Audio and images generated during the conference can be laid out in association with each section.

It is an example of composition of a conference system. It is an example of composition of a conference device, a conference server, and PC. It is an example of the screen displayed on a display device. It is an example of the data structure of meeting information. It is a flowchart which shows the procedure which records meeting information. It is a flowchart which shows the procedure which records the change of a display page. It is an example of the data structure of the minutes source information. It is an example of the data structure of the minutes source information. It is a flowchart which shows the procedure which produces | generates the minutes. It is a flowchart which shows the procedure of the layout processing of the minutes. A summary of the minutes to be generated. It is a flowchart which shows the procedure which produces | generates a summary text. It is a flowchart which shows the procedure of a meeting information recording process. It is a flowchart which shows the procedure of the minutes layout process. It is an example which produces | generates a difference image. It is a flowchart which shows the procedure of a meeting information recording process. It is an example of a display of a notification that prompts photographing. It is an example of a structure of an image information table. It is a flowchart which shows the procedure of the layout processing of the minutes.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration of a conference system according to the present embodiment.

  The conference system 100 includes a conference device 101, a conference server 102, and a PC 103. The conference apparatus 101 and the conference server 102 are connected via the network 104.

The conference apparatus 101 is arranged at a conference holding place such as a conference room, for example, on a conference table. The conference device 101 records a plurality of types of data (hereinafter referred to as “conference information”) such as voice and images generated during the conference.
The conference apparatus 101 detects a change point of a plurality of types of conference states and records it as a part of conference information. Then, the conference apparatus 101 transmits the conference information to the conference server 102.

Note that it is assumed that the conference apparatus 101 records a conference held in an office or a predetermined venue, for example. However, the “conference” of the present invention is not limited to a narrow-sense meeting held in an office, a predetermined venue, or the like, and may be a group that involves visual recognition / speaking behavior of a plurality of persons. For example, interviews and surveys are also included in the conference of the present invention.
In addition, the “minutes” of the present invention may be any information that can display conference information such as voices and images generated at the conference of the present invention in a displayable manner. Absent.

The conference server 102 is a general PC (Personal Computer) or a cloud server.
The conference server 102 receives the conference information from the conference device 101, analyzes and processes it, and creates the minutes. The conference server 102 recognizes voice data included in the conference information and generates text. In addition, text data is generated by recognizing image data included in the conference information.

The conference server 102 divides an entire conference into a plurality of sections. For this purpose, the conference server analyzes the conference information and detects the change point of the conference state. Then, based on the detected change point of the conference state, one entire conference is divided into a plurality of sections.
Here, “conference state” refers to various events that occur during the conference, such as the conference agenda, key points (described later), speakers, displayed materials, displayed material pages, and the like. is there. And it can define by the kind of several events in this way.
In addition, the “change point of the conference state” means the change of the conference state, that is, the change of the agenda in the conference, the occurrence of the main point, the change of the speaker, the change of the displayed material, the page of the displayed material Say when updates occur.
Thus, there are a plurality of types of change points. In addition, according to them, each section of the conference also has a plurality of types such as each agenda, each point, each speaker, each material, each material page, etc. (hereinafter, this section type is referred to as “section type”). Can be divided into The section type can be selected by the user.
The conference server 102 generates a minutes from information including the section and text described above. Then, distribute the created minutes.

In FIG. 1, the conference system 100 includes one conference device 101 and one conference server 102, but may be configured to include a plurality of conference devices 101 and conference servers 102.
In FIG. 1, the conference apparatus 101 and the conference server 102 are illustrated as separate apparatuses, but may be configured as a single apparatus having the functions of both apparatuses.

The PC 103 is a general PC or notebook PC.
The PC 103 can execute an application program and display materials to be referred to in the conference on the display. Further, a display may be projected by connecting a projector.
When the material displayed by the application program is changed in accordance with a user instruction or the like, the PC 103 transmits identification information of the displayed material to the conference apparatus 101. As the identification information, the file name of the material can be used. The PC 103 transmits display image data to the conference apparatus 101 when the document page displayed by the application program is changed in accordance with a user instruction or the like.

FIG. 2 is a block diagram illustrating a hardware configuration example of the conference apparatus 101, the conference server 102, and the PC 103 in FIG.
FIG. 2A is a hardware configuration example of the conference apparatus 101. FIG. 2B is a configuration example of the conference server 102 and a hardware configuration example of the PC 103.

  2A, the conference apparatus 101 includes a CPU 201, ROM 202, RAM 204, storage 205, input device 206, display device 207, external interface 208, camera device 209, and microphone device 210. These units can transmit and receive data to and from each other via the data bus 203. CPU is an abbreviation for Central Processing Unit. RAM is an abbreviation for Random Access Memory. ROM is an abbreviation for Read Only Memory.

The CPU 201 is a controller for controlling the entire conference apparatus. The CPU 201 activates an OS (Operating System) by a boot program stored in a ROM 202 which is a nonvolatile memory. The CPU 201 executes the controller program recorded in the storage 205 on the OS. The controller program is a program for controlling the entire conference apparatus. The CPU 201 controls each unit via a bus such as the data bus 203.
The RAM 204 operates as a temporary storage area such as the main memory or work area of the CPU 201.
The storage 205 is a non-volatile memory that can be read and written, and stores the aforementioned controller program. Further, the conference apparatus 101 records the conference information in the storage 205 until the conference information is transmitted to the conference server 102.

The input device 206 is an input device that includes a touch panel, hard keys, a mouse, and the like. The display device 207 is a display device such as an LCD. When the input device 206 receives an operation instruction from the user, the input device 206 transmits the instruction to the CPU 201.
The display device 207 displays the display image data generated by the CPU 201 on the screen. The CPU 201 determines an operation based on the instruction information received from the input device 206 and the display image data displayed on the display device 207. The CPU 201 controls the conference apparatus 101 according to the determination result, generates new display image data according to the operation content, and causes the display device 207 to display the display image data.

The external interface 208 transmits or receives various data to / from a separate external device via a network such as a local area network such as a LAN, a telephone line, or infrared rays.
The camera device 209 is a unit that can capture a moving image or an image. Specifically, a so-called digital camera is an example.
The microphone device 210 is means for converting the input sound into a digital signal. For example, the voice uttered by the user is acquired as voice data in the WAVE format or the like.
In this embodiment, the microphone device 210 is a microphone array having directivity around 360 °. If the microphone array is used, the direction of the utterance made around can be specified. An ID is assigned to each specified speech direction in order to identify each. The result of specifying the utterance direction is acquired as an ID.

  2B, the conference server 102 includes a CPU 251, a ROM 252, a RAM 254, a storage 255, an input device 256, a display device 257, and an external interface 258. Each unit can transmit / receive data to / from each other via the data bus 253.

  The CPU 251 is a controller for controlling the entire conference server 102. The CPU 251 starts up the OS by a boot program stored in the ROM 252 which is a nonvolatile memory. The CPU 251 implements each process of the conference server 102 by executing the conference server program stored in the storage 255 on the OS. The CPU 251 controls each unit via a bus such as the data bus 253.

The RAM 254 operates as a temporary storage area such as the main memory or work area of the CPU 251.
The storage 255 is a non-volatile memory that can be read and written, and stores the conference server program described above.

The input device 256 and the display device 257 are the same as the input device 206 and the display device 257 described with reference to FIG.
The external interface 258 is the same as the external interface 208 described with reference to FIG.

The hardware configuration of the PC 103 is the same as that of the conference server 102 described above.
In the PC 103, the storage 255 stores materials and application programs for displaying them.
The CPU 251 executes an application program on the OS, thereby interpreting the material and generating display image data for displaying it. Then, the display image data is displayed on the display device 257.

  Next, a user interface presented by the conference system 100 and referred to / operated by the user will be described.

FIG. 3 shows a display example of the display device 207 of the conference apparatus 101.
A screen 300 shown in FIG. 3A is a screen displayed before the start of the conference.

  A “start” button 301 is used by the user to instruct the conference apparatus 101 to start a conference. When the CPU 201 receives an instruction to the “start” button 301 via the input device 206, the CPU 201 displays a screen 310 shown in FIG. 3B on the display device 207. Then, the CPU 201 starts recording conference information.

  A screen 310 shown in FIG. 3B is a screen displayed during the conference.

  The “shoot” button 311 is used by the user to instruct the conference apparatus 101 to take a shooting request. When the CPU 201 receives an instruction to the “shoot” button 311 via the input device 206, the CPU 201 displays a screen 320 shown in FIG.

  The “Agenda” button 312 is for the user to instruct the conference apparatus 101 to change (start or end) the agenda. When the CPU 201 receives an instruction to the “Agenda” button 312 via the input device 206, the CPU 201 causes the display device 207 to display a screen 330 illustrated in FIG.

  The “end” button 313 is used by the user to instruct the conference apparatus 101 to end the conference. When the CPU 201 receives an instruction to the “end” button 313 via the input device 206, the CPU 201 causes the display device 207 to display a screen 340 shown in FIG.

  A screen 320 shown in FIG. 3C is a screen displayed when shooting. As shown in the figure, an image of a subject obtained by the camera device 209 is displayed. While viewing the video, the user can adjust the whiteboard or paper on which the characters are written to be within the shooting angle of view of the conference apparatus 101.

  When the CPU 201 receives an instruction to an arbitrary location on the screen 310 via the input device 206, the CPU 201 acquires image data of a subject photographed by the user with the camera device 209.

  The “OK” button 321 is used by the user to instruct the conference apparatus 101 to end shooting. When the CPU 201 receives an instruction to the “OK” button 321 via the input device 206, the CPU 201 displays a screen 310 shown in FIG.

A screen 330 illustrated in FIG. 3D is a screen displayed when an agenda change is instructed.
The text field 331 is for the user to register an agenda name in the conference apparatus 101. The user can enter a desired agenda name in the text field 331 via the input device 206.

The “start” button 332 is used by the user to instruct the conference apparatus 101 to start a new agenda.
The “end” button 333 is used by the user to instruct the conference apparatus 101 to end the current agenda. The names of the completed agendas are displayed in a list in the text area 334.
The “OK” button 335 is used by the user to instruct the conference apparatus 101 to end the agenda change. When the CPU 201 receives an instruction to the “OK” button 335 via the input device 206, the CPU 201 displays the screen 310 shown in FIG.

  A screen 340 shown in FIG. 3E is a screen displayed when the conference is ended.

  The text field 341 is used by the user to designate the destination of the minutes created by the conference system 100 to the conference apparatus 101. The user can input a desired destination in the text field 341 via the input device 206. For example, an e-mail address can be input as the transmission destination.

  The radio button 342 is used to select which section type is used to divide and summarize the meeting text obtained by integrating the utterance text and the entry text when generating the minutes. In this embodiment, the user can select any one section type.

The “OK” button 343 is used by the user to instruct the conference apparatus 101 to confirm the end of the conference. When the CPU 201 receives an instruction to the “OK” button 343 via the input device 206, the CPU 201 displays the screen 300 shown in FIG.
Then, the conference apparatus 101 ends the recording of the conference information and transmits the conference information to the conference server 102. Thereafter, the conference server 102 analyzes and processes the received conference information, generates a minutes, and transmits the minutes to the transmission destination.

Next, conference information recorded by the conference apparatus 101 will be described.
FIG. 4 shows a configuration example of conference information recorded in the storage 205 by the conference apparatus 101.

The audio information table 400 shown in FIG. 4A is a data table that records information about audio acquired by the conference apparatus 101 (hereinafter referred to as “audio information”). The conference apparatus 101 records a conversation during the conference and records voice data.
In the recording start time column 401, the recording start time (conference start time) is recorded. The recording end time column 402 records the recording end time (conference end time). In the audio data column 403, the file name of the audio data recorded in the storage 205 is recorded.

An image information table 410 illustrated in FIG. 4B is a data table that records information (hereinafter referred to as “image information”) related to an image captured and acquired by the conference apparatus 101. The conference apparatus 101 captures image data by capturing a subject and records it as a file in the storage 205.
The shooting time column 411 records the shooting time. In the image data string 412, the file name of the image data recorded in the storage 205 is recorded.

The agenda information table 420 shown in FIG. 4C is a data table that records information on the agenda recorded by the conference apparatus 101 (hereinafter referred to as “agenda information”).
The agenda start time column 421 records the start time of the agenda. The agenda end time column 422 records the end time of the agenda. The agenda name is recorded in the agenda name column 423.

The speaker information table 430 shown in FIG. 4D is a data table that records information related to speaker changes recorded by the conference apparatus 101 (hereinafter referred to as “speaker information”).
In the speech start time column 431, the time when the speaker starts speaking is recorded. The speech end time column 432 records the time when the speaker finished speaking. In the speaker ID column 433, information for identifying the speaker who has spoken is recorded. In the present embodiment, the ID indicating the speech direction acquired by the microphone device 210 of the conference apparatus 101 is recorded in the speaker ID column 433.

A material information table 440 shown in FIG. 4E is a data table that records information (hereinafter referred to as “material information”) related to material displayed by the PC 103 (hereinafter referred to as “display material”).
In the material start time column 441, the time when the PC 103 starts displaying the material is recorded. In the material end time column 442, the time when the PC 103 has finished displaying the material is recorded. The displayed material name is recorded in the material name column 453.
A material page information table 450 shown in FIG. 4F is a data table that records information related to a material page displayed by the PC 103 (hereinafter referred to as “material page information”).
In the document page start time column 451, the time when the PC 103 starts displaying the document page is recorded. The document page end time column 452 records the time when the PC 103 has finished displaying the document page. Information including the material name and page number of the displayed material is recorded in the material page number column 453.

Next, a process for recording the conference information of the conference apparatus 101 will be described.
5 and 6 are flowcharts showing the conference information recording process.
When the power is turned on by operating a power key (not shown) of the conference apparatus 101, the CPU 201 reads a controller program recorded in the storage 205. Then, the controller program is expanded in the RAM 204 and executed. As a result, the conference apparatus 101 can execute the conference information recording process. Further, the CPU 201 generates display image data of the screen 300 and causes the display device 207 to display the display image data.

First, in step S501, the CPU 201 determines whether an instruction to start a conference has been issued.
If an instruction is given to the “start” button 301, it is determined as YES. Then, the CPU 201 executes the processes from S502 to S515 (A to B in the figure) and the processes from S516 to S522 (C to D in the figure) in parallel. At this time, the CPU 201 generates display image data of the screen 310 and causes the display device 207 to display the display image data.
If no instruction is given to the “start” button 301 in S501, the determination is NO and the process proceeds to S524.

In step S <b> 502, the CPU 201 starts recording a conversation using the microphone device 210. Thereby, the CPU 201 acquires audio data.
Then, the CPU 201 records the current time in the recording start time column 401 of the audio information table 400.
In addition, the CPU 201 starts recording audio data as a file in the storage 205. The CPU 201 records the file name of the sound data in the sound data string 403 of the sound information table 400. The recording start time 401 is the time when the conference is started.

In step S <b> 503, the CPU 201 determines whether an instruction for requesting shooting has been issued. If an instruction is given to the “shoot” button 311, the determination is YES, and the process proceeds to S <b> 504.
At this time, the CPU 201 generates display image data of the screen 320 and displays it on the display device 207. If no instruction is given to the “shoot” button 311, the determination is NO and the process proceeds to S 507.

  In step S504, the CPU 201 determines whether a shooting instruction has been issued. If an instruction is given to an arbitrary location in the screen 320, it is determined YES and the process proceeds to S505. If there is no instruction in the screen 320, it is determined as NO and the process proceeds to S506.

In step S <b> 505, the CPU 201 acquires image data of a subject photographed by the user with the camera device 209. Further, the CPU 201 adds a new record to the image information table 410. Then, the current time is recorded in the shooting time column 411 of the new record.
Further, the CPU 201 records the image data as a file in the storage 205. Then, the file name of the image data is recorded in the image data string 412 of the image information table 410.

In step S <b> 506, the CPU 201 determines whether an instruction to end shooting has been issued. If an instruction is given to the “OK” button 321, the determination is YES, and the process proceeds to S 503.
At this time, the CPU 201 generates display image data of the screen 310 and causes the display device 207 to display the display image data.
If no instruction is given to the “OK” button 321, the determination is NO and the process proceeds to S 504.

In step S507, the CPU 201 determines whether an instruction to change the agenda has been issued. If an instruction is given to the “Agenda” button 312, the determination is YES and the process proceeds to S 508.
At this time, the CPU 201 generates display image data of the screen 330 and causes the display device 207 to display it.
If no instruction is given to the “Agenda” button 312, the determination is NO and the process proceeds to S 513.

In step S <b> 508, the CPU 201 determines whether an instruction to start an agenda has been issued. If an instruction has been given to the “start” button 332, the determination is YES and the process proceeds to S 509. If no instruction is given to the “start” button 332, it is determined as NO and the process proceeds to S510.
If there is an agenda that has been started in S509 and not ended in S511, the determination in S508 is NO.

In step S509, the CPU 201 starts a new agenda.
At this time, the CPU 201 adds a new record to the agenda information table 420. Then, the current time is recorded in the agenda start time column 421 of the added record.

In step S510, the CPU 201 determines whether an instruction to end the agenda has been issued. If an instruction to the “Finish” button 333 has been given, it is determined YES and the process proceeds to S511. If no instruction is given to the “end” button 333, it is determined as NO and the process proceeds to S512.
Note that if there is no agenda started in S509 and not ended in S511, the determination in S510 is NO.

In step S511, the CPU 201 ends the current agenda.
At this time, the CPU 201 records the current time in the agenda end time column 422 of the record added in S509 in the agenda information table 420. Further, the agenda name input in the text field 331 is recorded in the agenda name column 423.

In step S512, the CPU 201 determines whether an instruction to end the agenda change has been given. If an instruction is given to the “OK” button 335, it is determined as YES, and the process proceeds to S503.
At this time, the CPU 201 generates display image data of the screen 310 and causes the display device 207 to display the display image data.
If no instruction is given to the “OK” button 335, it is determined as NO and the process proceeds to S508.

In step S513, the CPU 201 determines whether an instruction to end the conference has been given. If an instruction is given to the “end” button 313, it is determined as YES and the process proceeds to S514.
At this time, the CPU 201 generates display image data of the screen 340 and displays it on the display device 207.
If no instruction is given to the “end” button 313, it is determined as NO, and the process proceeds to S503.

In step S <b> 514, the CPU 201 ends the conference recording by the microphone device 210.
At this time, the CPU 201 records the current time in the recording end time column 402 of the audio information table 400. If there is a record in the agenda information table 420 in which the end time is not recorded in the agenda end time column 422, the current time is recorded in the agenda end time column 422.

In step S515, the CPU 201 determines whether an instruction to finalize the conference has been issued. If the transmission destination is input in the text field 341 and the “OK” button 343 is instructed, the determination is YES, and the process proceeds to S523. If the transmission destination is not input in the text field 341 or if an instruction to the “OK” button 343 is not made, it is determined as NO, and the process of S515 is performed again.
Note that the transmission destination input in the text field 341 and the section type selected by the radio button 342 are recorded as part of the conference information.

On the other hand, regarding the process of C in the figure, in S516, the CPU 201 determines whether or not the speaker has changed.
For this reason, the CPU 201 reads out the latest record of the speaker information table 430, that is, the value of the speaker ID column 433 of the record having the latest value of the speech start time column 431. Also, the utterance direction is acquired by the microphone device 210, and the ID corresponding to the utterance direction is acquired. If these IDs are compared and they do not match, it is determined YES and the process proceeds to S517. Also, if there is no record in the speaker information table 430, that is, the first utterance, the determination is YES.
If they match, it is determined as NO and the process proceeds to S518.

In step S517, the CPU 201 records speaker changes.
At this time, the CPU 201 records the current time as the speech end time of the previous speaker in the speech end time column 432 of the latest record in the speaker information table 430.
Further, the CPU 201 adds a new record to the speaker information table 430. Then, the current time is recorded in the comment start time column 431 of the added record.

In step S <b> 518, the CPU 201 determines whether there is a change in the display material that is the material displayed by the PC 103.
If the file name of the displayed material is received from the PC 103 via the external interface 208, the determination is YES and the process proceeds to S519. If it has not received, it will determine with NO and will change to S520.

In S519, the CPU 201 records the change of the display material.
At this time, the CPU 201 records the current time as the display end time of the previous material in the latest record of the material information table 440, that is, the material end time column 442 of the record having the newest value in the material start time column 441. .
Further, the CPU 201 adds a new record to the material information table 440. Then, the current time is recorded in the material start time column 441 of the added record, and the file name of the received material is recorded in the material name column 453.

In step S520, the CPU 201 determines whether display image data is received.
If display image data is received from the PC 103 via the external interface 208, the determination is YES and the process proceeds to S521. If it has not received, it will determine with NO and will change to S522.

  In step S <b> 521, the CPU 201 records the change of the display material page based on the received display image data. Details of the processing in S521 will be described with reference to FIG.

  FIG. 6 is a flowchart illustrating a process in which the conference apparatus 101 records a change in the page of the display material.

In step S601, the CPU 201 determines whether there has been a change in display material.
If it determines with YES by S518, it will also determine YES here and will change to S602. If it is determined NO in S518, it is also determined as NO here, and the process proceeds to S605.

In step S <b> 602, the CPU 201 determines whether the display material is a material displayed for the first time in this meeting.
At this time, the CPU 201 refers to the material page information table 450 and checks whether there is a record in which the value of the material page number column 453 contains information including the name of the currently displayed material.
When there is no corresponding record, it determines with YES and changes to S603. This means that the displayed material is displayed for the first time in this conference.
If there is a corresponding record, it is determined as NO and the process proceeds to S604. This means that the application program of the PC 103 displays a certain material, displays another material, and then displays the previously displayed material again.

  In step S <b> 603, the CPU 201 temporarily stores the current page number as 1 in the RAM 204.

In step S <b> 604, the CPU 201 temporarily stores the current page number in the RAM 204 as the page number when the display material was last displayed.
At this time, the CPU 201 searches the document page information table 450 for a record including the name of the currently displayed material in the value of the material page number column 453. Among these records, the page number included in the document page number column 453 of the record whose document page start time column 451 is the latest time is acquired.

On the other hand, in step S605, the CPU 201 determines whether there is a change in the displayed page.
As will be described later, the display image data is stored in the storage 205. The CPU 201 reads from the storage 205 display image data corresponding to the record in the document page information table 450 and having the latest record page start time column 451.
Then, the CPU 201 calculates an image difference with display image data received via the external interface 208. If there is a difference pixel more than a predetermined threshold, it is determined YES, that is, the displayed page has been changed, and the process proceeds to S606. If the pixel having the difference is less than the predetermined threshold, it is determined as NO, and the process of the flowchart of FIG. 6 is terminated.

In step S606, the CPU 201 determines whether the currently displayed page is a new page.
The CPU 201 selects a record that includes the name of the currently displayed material in the value of the material page number column 453 in the material page information table 450 and whose material page start time column 451 is the second newest time. To do. Then, the display image data corresponding to the record is read from the storage 205.
Then, the CPU 201 calculates an image difference with display image data received via the external interface 208. If the difference pixel is less than the predetermined threshold, it is determined NO, that is, the previous page is displayed, and the process proceeds to S607. If the difference pixel is equal to or greater than a predetermined threshold, it is determined YES, that is, the next new page is displayed, and the process proceeds to S608.

  In step S <b> 607, the CPU 201 temporarily stores the current page number in the RAM 204 as it is. That is, the page number included in the material page number column 453 of the record selected in S606 is stored.

  In step S <b> 608, the CPU 201 temporarily stores the current page number as the next page number in the RAM 204. That is, 1 is added to the page number included in the material page number column 453 of the record referred to in S605, and the value is stored.

In step S609, the CPU 201 records a page change.
At this time, the CPU 201 acquires the latest record from the material page information table 450, that is, the latest record in the material page start time column 451. Then, the current time is recorded as the page display end time in the document page end time column 452 of the record.
In addition, the CPU 201 adds a new record to the material page information table 450. Then, the current time is stored in the material page start time column 451 of the added record. Further, information including the file name of the currently displayed material received via the external interface 208 and the page number temporarily stored in the RAM 204 is stored in the material page number column 453 of the added record.

In step S <b> 610, the CPU 201 stores display image data received via the external interface 208 in the storage 205.
At this time, the display image data can be referred to in S605 and S606 by using the value of the document page number column 453 of the record added in S609 as the file name of the display image data in the storage 205.

  Returning to FIG. 5, in step S <b> 522, the CPU 201 determines whether an instruction to end the conference has been issued. This is the same processing as S513.

  In step S <b> 523, the CPU 201 transmits the conference information recorded in the storage 205 in the above processing to the conference server 102 via the external interface 208. After the transmission, the conference information may be deleted from the storage 205. Further, the CPU 201 generates display image data of the screen 300 and causes the display device 207 to display the display image data.

In step S524, the CPU 201 determines whether an instruction to turn off the power has been given.
For example, it is determined that an instruction to turn off the power is given by pressing the power key of the conference apparatus 101 or detecting a specific word (command) in the user's utterance.
If an instruction to turn off the power has been given, the determination is YES and the process is terminated. If no instruction is given, it is determined as NO, and the process proceeds to step 501.

Next, the minutes source information generated by the conference server 102 analyzing and processing the conference information received from the conference device 101 will be described.
7 and 8 show configuration examples of the minutes source information recorded in the storage 255 by the conference server 102. FIG.

The utterance text information table 700 shown in FIG. 7A is a data table that records information related to the speech recognition result for the speech data included in the conference information (hereinafter referred to as “utterance text information”). The utterance text information table is generated as a record for each utterance when voice data is analyzed and a user's utterance is specified.
The utterance time column 701 records the time when the utterance occurred (hereinafter referred to as “utterance time”). The utterance text column 702 records the utterance text obtained by voice recognition of the utterance. The specification of the utterance and the utterance time will be described later.

An entry text information table 710 shown in FIG. 7B is a data table for recording information about a character recognition result for image data included in the conference information (hereinafter referred to as “entry text information”). The entry text information table 710 is generated as a record for each entry when the image data is analyzed and the entry by the user is specified.
The entry time column 711 records the time at which entry occurred (hereinafter referred to as “entry time”). The entry text column 712 records entry text obtained by character recognition of image data. The specification of entry and entry time will be described later.

The conference text information table 720 shown in FIG. 7C is a data table for recording information (hereinafter referred to as “conference text information”) regarding text generated in the conference (hereinafter referred to as “conference text information”). The meeting text information is generated by integrating the utterance text information shown in FIG. 7A and the entry text information shown in FIG.
The occurrence time column 721 records the time when the meeting text information is generated, and records the time of the utterance time 701 or the entry time 711.
The conference text column 722 records the conference text, and records the text of the utterance text column 702 or the entry text column 712.
The classification column 723 records whether the record was utterance text information or entry text information before integration. If it is utterance text information, 0 is recorded, and if it is entry text information, 1 is recorded.
The point column 724 records whether or not the meeting text in the meeting text column 722 of the record is a point. Here, the main points indicate main contents in the conference such as matters determined in the conference and action items generated (requests for work performed on participants in the conference). If it is the main point, 1 is recorded, otherwise 0 is recorded.
The records in the conference text information table 720 are sorted in ascending order (in the order in which they occurred) by the value in the occurrence time column 721.

The point information table 730 shown in FIG. 7D is a data table for recording information about points (hereinafter referred to as “point information”).
The start time of the section is recorded in the main point start time column 731. Further, the end time of the section is recorded in the main point end time column 732.
Here, in this embodiment, the “section” is defined as a period from the occurrence of “a certain point” to the occurrence of “the next another point”. Therefore, the main point start time column 731 is the occurrence time of the “certain point”. Further, the main point end time column 732 is an occurrence time of the “next main point”.
Note that the recording start time (conference start time) is recorded in the main point start time column 731 of the main point that occurred first. The contents of the main points are recorded in the main point column 733.

The summary information table 800 illustrated in FIG. 8 is a data table in which information (hereinafter referred to as “summary information”) that summarizes the conference text illustrated in FIG. 7C is recorded. The summary text is generated by summarizing the conference text in the conference text column 722 of the conference text information table 720 for each section of the type specified by the user operating the radio button 342.
In the section name column 801, the section name is recorded. The section name will be described later.
The generated summary text is recorded in the summary text column 802.

Next, a process of generating the minutes by the conference server 102 will be described using the flowchart of FIG.
First, when the conference server 102 is activated, the CPU 251 reads a conference server program recorded in the storage 255. Then, the conference server program is expanded in the RAM 254 and executed. Thereby, the conference server 102 can execute the minutes generation process.

In step S901 in FIG. 9A, the CPU 251 determines whether conference information has been received.
If the conference information is received from the conference apparatus 101 via the external interface 258, it is determined as YES and the process proceeds to S902. If the conference information is not received from the conference apparatus 101 via the external interface 258, it is determined as NO and the process proceeds to S909.

In step S <b> 902, the CPU 251 performs voice recognition on the voice data included in the received conference information, and acquires the utterance text.
Here, the CPU 251 scans the voice data from the head, performs voice recognition for each utterance section, and acquires the utterance text.
Note that the utterance section refers to a section between a certain silent section in the audio data and the next silent section. The silent section is detected based on, for example, a state in which the sound pressure of the audio data is below a threshold value for a certain period of time.

Further, the CPU 251 calculates the utterance time of each utterance section from the recording start time in the recording start time column 401 of the conference information voice information table 400 and the elapsed time from the beginning of the voice data of each utterance section. Then, a record of each utterance section is generated in the utterance text information table 700. Furthermore, the utterance time and the utterance text of each utterance section are recorded in the utterance time column 701 and the utterance text column 702, respectively.
The entry text information table 710 shown in FIG. 7A is created by the processing of S902.

In step S <b> 903, the CPU 251 performs character recognition on the image data included in the received conference information, and acquires entry text.
Here, the CPU 251 sequentially scans the records in the image information table 410 included in the conference information, and performs the following processing.
First, the records of the image information table 410 are sorted in ascending order (in order of shooting) by the value of the shooting time column 411. The CPU 251 obtains an image difference between the image data indicated by the image data sequence 412 of the currently referenced record and the image data indicated by the image data sequence 412 of the previous record. The image difference can be regarded as a partial image including characters entered by the user from the shooting corresponding to the previous record to the shooting corresponding to the currently referenced record. Therefore, character recognition is performed on the partial image to obtain entry text.

In addition, the shooting time in the shooting time column 411 of the record that is currently referred to is set as the time when the image difference occurs, that is, the entry time when the user has entered.
A record is generated in the entry text information table 710 for each entry time and entry text obtained in this way. The entry time and entry text are recorded in the entry time column 711 and entry text column 712, respectively.
By the process of S903, the utterance text information table 700 shown in FIG. 7B is created.

In step S904, the CPU 251 integrates the utterance text and the entry text to generate a conference text.
That is, the CPU 251 integrates the utterance text information table 700 generated in S902 and the entry text information table 710 generated in S903 to generate a conference text information table 720.

Here, the CPU 251 generates the conference text information table 720 by performing the following processing.
First, the CPU 251 adds a record corresponding to each record included in the utterance text information table 700 to the conference text information table 720.
At this time, the CPU 251 records the utterance time in the utterance time column 701 in the occurrence time column 721 as the time when the conference text occurs. Also, the utterance text in the utterance text column 702 is recorded in the conference text column 722 as the conference text. In the division column 723, 0 indicating that the original data was utterance text information is recorded.

In addition, the CPU 251 adds a record corresponding to each record included in the entry text information table 710 to the conference text information table 720.
At this time, the CPU 251 records the entry time in the entry time column 711 in the occurrence time column 721 as the time when the meeting text occurs. The entry text in the entry text column 712 is recorded in the conference text column 722 as the conference text. In the division column 723, 1 indicating that the original data was entry text information is recorded.
Then, the CPU 251 sorts the records added as described above in ascending order (in the order in which they occurred) by the value in the occurrence time column 721.

In S905, the CPU 251 extracts the main points from the conference text generated in S904.
Here, the details of the point extraction process will be described with reference to FIG.
In S921 in FIG. 9B, the CPU 251 selects a record in the conference text information table 720 to be processed. When S921 is executed for the first time, the top record of the conference text information table 720 is selected. From the second time onward, the record next to the currently selected record is selected.

In S922, the CPU 251 determines whether or not the main point extraction processing has been performed for all the records in the conference text information table 720.
If there is no next record in S921 and it cannot be selected, it is determined as YES and the point extraction process is terminated.
If the next record can be selected in S921, the determination is NO and the process proceeds to S923.

In S923, the CPU 251 determines whether or not the meeting text of the currently selected record is the main point.
For example, if the meeting text includes a specific keyword determined in advance, it is determined that it is the main point. If it is a point, 1 is recorded in the point column 724 of the currently selected record. Otherwise, 0 is recorded.
With the above processing, the conference text information table 720 shown in FIG. 7C is created.

In S924, the CPU 251 determines whether or not the meeting text of the currently selected record is the main point.
When it determines with it being a main point in S923, it determines with YES and changes to S925. If it is determined in S923 that it is not the main point, it is determined as NO and the process proceeds to S921.

In S925, the CPU 251 stores that the main point has occurred.
If there is a record in the gist information table 730, the CPU 251 acquires the latest record, that is, the record with the latest value in the gist start time column 731. Then, the value of the record in the main point end time column 732 of the record is acquired as the main point start time.
When there is no record in the main point information table 730, the CPU 251 acquires the value of the recording start time column 401 of the audio information table 400 (that is, the conference start time) as the main point start time.
Then, the CPU 251 adds a new record to the point information table 730 and records the acquired point start time in the point start time column 731 of the added record. Further, the value of the occurrence time column 721 of the record of the currently selected conference text information table 720 is recorded in the main point end time column 732. In the gist column 733, the value of the conference text column 722 of the record of the currently selected conference text information table 720 is recorded as the content of the gist.
Through the above processing, the main point information table 730 shown in FIG. 7D is created.

Returning to FIG. 9A, in S906, the CPU 251 performs a summarization process for summarizing the conference text generated in S904.
Here, the details of the summary process will be described with reference to FIG.

In S941, the CPU 251 selects a section type to be used for the summary of the conference text.
Here, the CPU 251 refers to and selects the section type information selected by the user included in the conference information. Based on this, a table to be referred to is selected from the agenda information table 420, the speaker information table 430, the material information table 440, the material page information table 450, and the main point information table 730.

  That is, if the section type selected by the user is “per agenda”, the agenda information table 420 is selected. If the section type selected by the user is “for each speaker”, the speaker information table 430 is selected. If the section type selected by the user is “for each material”, the material information table 440 is selected. If the section type selected by the user is “every document page”, the document page information table 450 is selected. If the section type selected by the user is “for each key point”, the key point information table 730 is selected.

Here, the agenda information table 420, the speaker information table 430, the material information table 440, the material page information table 450, and the main point information table 730 are collectively referred to as “section information table”.
Further, the agenda start time column 421, the speech start time column 431, the material start time column 441, the material page start time column 451, and the main point start time column 731 are collectively referred to as “section start time column”.
Further, the agenda end time column 422, the speech end time column 432, the material end time column 442, the material page end time column 452, and the main point end time column 732 are collectively referred to as “section end time column”.
Further, the agenda name column 423, the speaker column 433, the material name column 453, the material page number column 453, and the main point column 733 are collectively referred to as “section name column”.

In S942, the CPU 251 selects a record in the section information table to be processed.
When executing S942 for the first time, the top record of the section information table is selected. From the second time onward, the record next to the currently selected record is selected.

In S943, the CPU 251 determines whether or not all the records in the section information table have been processed.
In S942, if there is no next record and the record cannot be selected, it is determined as YES, and the summary process is terminated. In S942, when the next record can be selected, it is determined as NO, and the process proceeds to S944.

In S944, the CPU 251 acquires the conference text in the section of the currently selected record.
Here, the CPU 251 records from the conference text information table 720 that the time in the occurrence time column 721 corresponds to the time from the time in the section start time column of the currently selected record to the time in the section end time column. To extract all. Then, the conference text in the conference text column 722 of one or more extracted records is set as a summary target.

In S945, the CPU 251 summarizes the meeting text acquired in S944.
Details of the summary text generation processing will be described with reference to FIG.

  FIG. 12A is a flowchart illustrating an example of the summary text generation process. The process of this flowchart is executed for a plurality of records in the conference text information table 720.

In step S <b> 1201, the CPU 251 refers to all input records in the conference text information table 720 and calculates the importance of each word appearing in the conference text in the conference text column 722.
The importance of each word can be calculated based on the appearance frequency of each word, for example.

In S1202, the CPU 251 calculates the importance of each conference text in the conference text column 722 of each input record.
The importance of the conference text is determined by referring to the importance of each word calculated in S1201 and calculating the total value of the importance of the words included in each conference text.

In step S1203, the CPU 251 extracts meeting text having an importance level equal to or higher than a threshold value. These meeting texts are combined to generate a summary text.
When the summary text is generated, the process returns to the process caller.

In a meeting, a point of entry or an important opinion may be written on an entry medium such as a whiteboard. Since the contents thus described are often important as agendas, they may be preferentially reflected in the summary text as entry text.
In this case, for example, in S1203, the CPU 251 confirms the value of the classification column 723 of the input record. If the value of the classification column 723 is 1 (original information), the meeting text 722 of the record is extracted as a part of the summary text regardless of whether the importance is equal to or higher than the threshold value.

Alternatively, another process may be executed to preferentially reflect the entry text in the summary text. FIG. 12B is a flowchart showing another example of the summary text generation process.
Similar to the flowchart of FIG. 12A, the processing of the flowchart of FIG. 12B is executed for a plurality of records in the conference text information table 720.

  In S1211, the CPU 251 obtains a list of words included in the conference text column 722 (hereinafter referred to as “entry word list”) for records whose classification column 723 is 1 (originally entry information) among the input records. create.

In S1212, the CPU 251 performs the same process as S1201.
In S1213, the CPU 251 performs the same process as in S1202.
“0130”
In S1214, the CPU 251 updates the importance of the conference text calculated in S1213 with reference to the entry word list created in S1211.
That is, when the conference text 722 of each input record includes a word in the entry word list, a bias is added to the importance.

  In S1215, the CPU 251 performs the same process as in S1203.

  Alternatively, in S1214, the CPU 251 may assign the maximum importance level when the target meeting text includes a word in the entry word list. In this case, in S1203, the meeting text is extracted as a part of the summary text.

Then, the CPU 251 adds a new record to the summary information table 800. Then, the generated summary text is recorded in the summary text column 802 of the added record. In addition, the value of the section name column of the section information table currently referred to is recorded in the section name column 801 of the added record.
Through the above processing, the summary information table 800 shown in FIG. 8 is created.

  Returning to FIG. 9A, in S907, the CPU 251 executes the layout process of the minutes based on the minutes source information and the meeting information obtained as described above.

Here, the layout process of the minutes will be described with reference to FIGS. 10 and 11.
FIG. 10 is a flowchart showing the minutes layout process.

  In S1001, the CPU 251 selects a record of the summary information table 800 to be processed. When S1001 is executed for the first time, the first record of the summary information table 800 is selected. From the second time onward, the record next to the currently selected record is selected.

In step S <b> 1002, the CPU 251 determines whether all the records in the summary information table 800 have been processed.
In S1002, if there is no next record and it cannot be selected, it is determined as YES and the layout processing is terminated. In S1002, when the next record can be selected, it is determined as NO, and the process proceeds to S1003.

  In S1003, the CPU 251 acquires the section start time 731 and the section end time 732 in the section in which the summary generation of the selected record has been executed, from the point information table 730 in FIG. Specifically, the section start time and section end time of the section that is the subject of summarization in S944 are acquired.

  In step S1004, the CPU 251 acquires the image data 412 in the section in which the summary of the record with the shooting time selected is executed from the image information table 410 in FIG. That is, the CPU 251 specifies a record in the image information table 410 that corresponds to the shooting time string 411 between the section start time and the section end time acquired in S1003. Then, the image data is acquired with reference to the file name included in the image data string 412 of the specified record.

  In step S1005, the CPU 251 creates a layout on the screen by associating the summary text of the record selected in step S1001 with the image data acquired in step S1004.

FIG. 11 is a diagram showing an example of the layout of minutes.
The minutes 1100 are composed of a plurality of agenda pages 1110 including text data and image data. Each agenda page 1110 includes a layout 1120 (1121) including summary text and image data acquired for each section in which the summary is executed.
In each layout 1120 (1121), the text 1130 (1134) of the section name column 801 of the record selected in S1001, the summary text 1131 (1135) corresponding thereto, and the image data 1132 (1136) are displayed. Further, the file name 1133 (1137) of the image data may be displayed.
In FIG. 11, the image data 1132 (1136) is displayed at the end of the summary text 1131 (1135), but this layout is merely an example. Each layout of the text 1130 (1134), the summary text 1131 (1135), and the image data 1132 (1136) is not limited to this example. For example, summary text and image data may be laid out side by side.

FIG. 11 will be described in detail along the flowchart of FIG.
Here, an example of layout for each agenda will be described. Hereinafter, description will be made using the summary information table 800 (FIG. 8), the image information table 410 (FIG. 4B), and the agenda information table 420 (FIG. 4C).

First, in S1001, the CPU 251 selects one record from the summary information table.
Here, a record with the section name 801 “about budget for the next year” in the summary information table 800 is selected.

In S1002, the CPU 251 determines whether or not all sections have been processed.
Here, since all the sections of the summary information table 800 have not yet been processed, the determination is no.

In S1003, the CPU 251 acquires the start time and end time of the section.
Here, in the agenda information table 420, the agenda start time 421 (2017/08/09 13:02:27) and the agenda end time 422 (2017/08/09 13) of the record whose agenda name 423 is “about the budget for the next fiscal year”. : 31: 55).

In step S <b> 1004, the CPU 251 selects and acquires image data whose shooting time corresponds to the end time from the agenda start time.
Here, the CPU 251 acquires image data (“image1.jpg”) that falls within the range of the agenda start time 421 to the agenda end time 422 in the image information table 410.

In step S1005, the CPU 251 creates a layout of the summary text of the record being selected and the acquired image data.
Here, the CPU 251 first displays the section name 801 (“about budget for next year”) in the selected record as text 1130.
Subsequently, the summary text 802 (“the budget for the next year is...)” In the selected record is displayed as the summary text 1131.
Subsequently, the image and file name (“image1.jpg”) of the acquired image data 412 are displayed as an image 1132 and a file name 1133, respectively.
Output.

In S1004, if there are a plurality of corresponding images, the CPU 251 displays the image data 1132 and the file name 1133 in succession.
With the above processing, the layout 1120 in which the summary text 1131 and the image data 1132 are combined can be displayed with respect to the “next budget budget” agenda.

Returning to S1001, the CPU 251 selects the next record from the summary information table.
Here, a record whose section name 801 is “consultation of role assignment” in the summary information table 800 is selected.
Thereafter, by repeatedly executing the same processing, it is possible to create a layout 1121 that combines the summary text 1135 and the image data 1136 for the “role-sharing consultation” agenda.

  If there are no more records to be selected from the summary information table in S1001, the determination in S1002 is YES. Then, the minutes layout process ends, and the minutes 1100 are generated.

Returning to FIG. 9A again, in S908, the CPU 251 transmits the minutes to the transmission destination included in the conference information. As a transmission method, for example, it can be transmitted by electronic mail.
Here, the CPU 251 attaches the minutes 1100 as an attached file to the e-mail and transmits the e-mail.

In step S909, the CPU 251 determines whether termination has been instructed.
For example, the user can instruct the termination of the conference server 102 from a separate PC via the external interface 258.
If termination is instructed, it is determined as YES, and the process is terminated. If termination is not instructed, it is determined as NO, and the process proceeds to S901.

  As described above, according to the first embodiment, meeting information can be divided into a plurality of sections, and summary text and image data can be associated with each section to generate a laid-out minutes.

In the first embodiment, the user manually photographed the whiteboard or paper that was written during the meeting using the camera device 209.
On the other hand, in the second embodiment, the CPU 201 automatically captures the subject by setting the camera device 209 facing the whiteboard or paper. Below, the difference with Example 1 is mainly demonstrated.

FIG. 13A is a flowchart illustrating a process for recording conference information according to the second embodiment.
The process from E to F in the figure, which is a difference from the first embodiment, will be described later with reference to FIG. The processes from E to F in the figure are executed in parallel with the processes from A to B in the figure and from C to D in the figure. The process from A to B in the figure and the process from C to D in the figure are the same as the process described in FIG.

FIG. 13B is a flowchart illustrating automatic shooting processing according to the second embodiment.
The timing for starting automatic shooting is when the user finishes writing on the whiteboard or paper (when a person or hand disappears from the video), or when a change point in the conference state such as an agenda change is detected. .

In step S <b> 1301, the CPU 201 determines whether a moving object such as a person or a hand is included in the video acquired via the camera device 209.
When it is included, it determines with YES and changes to S1302. When not included, it determines with NO and changes to S1303.

  In step S1302, the CPU 201 turns on the moving object detection flag.

In S1303, the CPU 201 determines whether or not the moving object detection flag is ON.
If the moving object detection flag is ON, the determination is YES, and the process proceeds to S1304. When the moving object detection flag is not ON, it is determined as NO, and the process proceeds to S1306.

  In step S <b> 1304, the CPU 201 captures an image of the subject via the camera device 209 and acquires image data.

In step S1305, the CPU 201 turns off the moving object detection flag.
Then, the CPU 201 acquires image data of a subject photographed via the camera device 209.

In step S1306, the CPU 201 determines whether a meeting state change point has been detected.
Here, when the end of the agenda is detected in S510, when the change of the speaker is detected in S516, or when the change of the display material is detected in S518, it is determined that the meeting state change point has occurred.
If a meeting state change point is detected, the determination is YES and the process proceeds to S1307. When the change point is not detected, it determines with NO and changes to S1308.

  In step S <b> 1307, the CPU 251 captures a subject via the camera device 209 and acquires subject image data.

In step S1308, the CPU 201 determines whether an instruction to turn off the power has been issued.
If an instruction is given to the power key of the conference apparatus 101, the determination is YES, and the process ends. If no instruction is given to the power key of the conference apparatus 101, it is determined as NO and the process proceeds to S1301.

  As described above, according to the second embodiment, it is possible to automatically shoot a whiteboard or paper written by the user during the meeting, so that the user does not forget to shoot and records image data more accurately. Can be left in the minutes.

Writing on the whiteboard during the meeting is added as appropriate as the discussion proceeds. For this reason, the image captured by the camera device 209 often includes writing included in the previously captured image and writing newly added this time.
Therefore, in the third embodiment, only newly added writing is generated as a difference image and laid out in the minutes. Below, the difference with Example 1 is mainly demonstrated.

FIG. 14 is a flowchart illustrating the minutes layout process according to the third embodiment.
In step S1401, the CPU 251 generates a difference image from the image data captured immediately before the image data acquired in step S1004. A method for generating the difference image will be described with reference to FIG.

In the example of FIG. 15, (a) is the image data acquired in S1004, and (b) is the image data photographed immediately before the acquired image data.
First, each image is scanned in units of pixels, and pixels whose pixel value difference is equal to or greater than a threshold value are specified. Then, by obtaining a circumscribed rectangle that includes all the specified pixels in the image data of FIG. 15A, a differential image as shown in FIG. 15C is obtained.
In S1402, the CPU 251 causes the summary text and the difference image generated in S1401 as image data to be laid out in association with each other.

  As described above, according to the third embodiment, since only newly added writing is generated as a difference image and the minutes are generated, it is possible to reduce the overlap of the contents of the minutes.

In the first embodiment, the corresponding image data between the start time and the end time of the section for which the summary is to be created is laid out together with the summary text, and the minutes are generated. Therefore, the image data is not displayed together with the summary text unless shooting is performed within the section to be summarized, such as when the user forgets to shoot.
Therefore, in the fourth embodiment, a message prompting the user to shoot is notified at the end of the section to be summarized. Below, the difference with Example 1 is mainly demonstrated.

FIG. 16A is a flowchart illustrating a process for recording conference information according to the fourth embodiment.
The process from G to H in the figure, which is a difference from the first embodiment, will be described later with reference to FIG. The processes from G to H in the figure are executed in parallel with the processes from A to B in the figure and from C to D in the figure. The processes from A to B in the figure and the processes from C to D in the figure are the same as the processes described in FIG.

In the flowchart of FIG. 16B, the user is notified of a request for shooting.
First, in step S1601, the CPU 201 determines whether a meeting state change point has been detected.
Here, when the end of the agenda is detected at S510, when the change of the speaker is detected at S516, or when the change of the display material is detected at S518, it is determined that the meeting state change point has occurred.
If a meeting state change point is detected, the determination is YES and the process proceeds to S1602. When the change point is not detected, it determines with NO and changes to S1603.

In step S1602, the CPU 201 notifies a message for prompting the user to shoot. FIG. 17 is a display example of a message prompting the user to shoot on the display device 207 of the conference apparatus 101.
In this case, a message notification 1700 such as “Please execute shooting of the completed agenda” is displayed at the top of the screen 310 (FIG. 3B) displayed during the conference.

In step S <b> 1603, the CPU 201 determines whether a notification that prompts shooting is displayed on the display device 207 of the conference apparatus 101.
If the notification is being made, the determination is YES, and the process proceeds to S1604. When not informing, it judges with NO and changes to S1609.

In step S <b> 1604, the CPU 201 determines whether the user has selected the message notification 1700 displayed on the display device 207. That is, it is confirmed whether or not the user wants to perform shooting for the already completed section.
If the user selects the message notification 1700, the determination is YES, and the process proceeds to S1605. At this time, the CPU 201 generates display image data of the screen 320 and displays it on the display device 207. When the user does not select the message notification 1700, it is determined as NO, and the process proceeds to S1609.

In step S1605, the CPU 201 determines whether shooting is instructed.
If an instruction has been given to an arbitrary location on the screen 320, it is determined YES and the process proceeds to S1606. If no instruction is given on the screen 320, it is determined as NO and the process proceeds to S1609.

In step S <b> 1606, the CPU 201 captures an image of the subject via the camera device 209, acquires image data, and records the image data in the storage 205.
FIG. 18 shows a configuration example of the image information table 1800 that the conference apparatus 101 records in the storage 205 in this embodiment. As a difference from the image information table of the first embodiment (FIG. 4B), the section name detected in S1601 is recorded in the related section name column 1801 in order to associate the image data with the section.

In step S <b> 1607, the CPU 201 determines whether an instruction to end shooting has been issued.
If an instruction is given to the “OK” button 321, the determination is YES, and the process proceeds to S 1608. At this time, the CPU 201 generates display image data of the screen 310 and causes the display device 207 to display the display image data. If no instruction is given to the “OK” button 321, the determination is NO and the process proceeds to S 1605.

  In step S <b> 1608, the CPU 201 hides the notification message 1700 displayed on the screen 310.

In step S1609, the CPU 201 determines whether an instruction to turn off the power has been given.
If an instruction is given to the power key of the conference apparatus 101, the determination is YES, and the process ends. If no instruction is given to the power key of the conference apparatus 101, it is determined as NO and the process proceeds to S1601.

Next, a process in which the conference server 102 generates the minutes will be described.
FIG. 19 is a flowchart illustrating a minutes layout process according to the fourth embodiment.

In the first embodiment, image data corresponding to the shooting time is extracted from the start time to the end time of the section to be summarized such as an agenda. However, for example, after the agenda is finished and the user moves to the next agenda, when the user takes a picture of the previous agenda, the photographed image data cannot be extracted corresponding to the agenda only with the time information.
Therefore, in the flowchart shown in FIG. 19, image data is acquired with reference to not only time information but also a related section name string 1801 in the image information table 1800.

  In S1901, the CPU 251 refers to the related section name column 1801 in the image information table 1800, specifies a record that matches the section name of the record selected in S1001, and acquires the image data.

Here, in the fourth embodiment, the process for generating the minutes shown in FIG. 11 will be described with reference to the flowchart of FIG.
Here, as in the first embodiment, an example of layout for each agenda will be described. Hereinafter, the summary information table 800, the image information table 1800, and the agenda information table 420 will be described.

First, in S1001, the CPU 251 selects a record from the summary information table.
Here, the CPU 251 selects a record whose section name 801 in the summary information table 800 is “about budget for the next year”.

In S1002, the CPU 251 determines whether or not all sections have been processed.
Here, since all the records of the summary information table 800 have not been processed yet, the determination is NO.

In S1003, the CPU 251 acquires the start time and end time of the section.
Here, in the agenda information table 420, the agenda start time 421 (2017/08/09 13:02:27) and the agenda end time (2017/08/09 13: 31:55).

In step S1004, the CPU 251 acquires image data whose shooting time corresponds to the end time from the agenda start time.
Here, referring to the imaging time column 411 of the image information table 1800, since there is no record in which the imaging time is between the agenda start time and end time, the CPU 251 does not acquire image data.

In step S1901, the CPU 251 acquires image data that matches the section name of the record being referred to.
Here, the CPU 251 acquires the image data “image1.jpg” of the record that matches the section name “about the budget for the next year” of the record being selected from the related section name column 1801 of the image information table 1800.
Thereafter, by executing the same processing as in the first embodiment, the minutes 1100 are generated.

  As described above, according to the fourth embodiment, the user can also perform shooting for a section that has already been completed. For this reason, it is possible to reduce the fact that the corresponding image is not laid out even if the user forgets to take a picture in a timely manner.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device.
The present invention is not limited to the above-described embodiments, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and these are excluded from the scope of the present invention. It is not a thing. That is, the present invention includes all configurations in which the above-described embodiments and modifications thereof are combined.

Claims (15)

First acquisition means for acquiring text generated from audio data in the conference information generated in the conference, and a first time when the audio data is generated;
Second acquisition means for acquiring image data in the meeting information and a second time when the image data is generated;
First recording means for recording the text and the first time;
Second recording means for recording the image data and the second time;
Dividing means for dividing the conference into a plurality of sections;
First selection means for selecting, from the text recorded in the first recording means, text generated from the start time to the end time of the section;
Second selection means for selecting image data generated from the start time to the end time of the section from the image data recorded in the second recording means;
A minutes generation apparatus comprising: a creation unit that creates a layout by associating the text selected by the first selection unit with the image data selected by the second selection unit .   2. The minutes generation apparatus according to claim 1, wherein the text is a summary text obtained by summarizing the utterance text generated from the voice data. A third acquisition means for acquiring entry text generated from the image data in the meeting information and a third time when the image data is generated;
The first recording means integrates the text acquired by the first acquisition means and the entry text, and integrates and records the first time and the third time. The minutes generating device according to claim 1 or 2 characterized by things.   The first selecting means further selects an entry text generated from a start time to an end time of the section from the entry text recorded in the first recording means. The minutes generating device according to claim 3.   5. The minutes generation apparatus according to claim 1, wherein the dividing unit divides the meeting into a plurality of sections based on a meeting state.   6. The minutes generating apparatus according to claim 5, wherein a plurality of types are defined in the meeting state.   7. The minutes generation apparatus according to claim 6, further comprising an input unit that accepts selection of one type by the user from among the plurality of types.   8. The apparatus according to claim 1, further comprising a detecting unit configured to detect a change point of the conference state, wherein the dividing unit divides the conference into a plurality of sections based on the change point of the conference state. The minutes generating device according to item 1.   The minutes generating device according to any one of claims 1 to 8, wherein the photographing unit acquires the image data.   10. The minutes generation apparatus according to claim 9, wherein the photographing unit automatically acquires the image data.   The entry text acquired by the third acquisition means includes first image data acquired from the conference information, and second acquired from the conference information before acquiring the first image data. The minutes generation apparatus according to any one of claims 3 to 10, wherein the minutes are generated from a difference from the image data. The second recording means records information for identifying a section associated with the recorded image data,
The said 2nd selection means selects the image data matched with the said area from the said image data recorded on the said 2nd recording means, Any one of Claim 1 to 11 characterized by the above-mentioned. The minutes generating device according to item 1.   13. The minutes generation apparatus according to claim 8, further comprising display means for displaying a message to a user when the detection means detects a change point of the conference state. A first acquisition step of acquiring text generated from audio data in conference information generated in the conference, and a first time when the audio data is generated;
A second acquisition step of acquiring image data in the conference information and a second time when the image data is generated;
A first recording step for recording the text and the first time;
A second recording step for recording the image data and the second time;
A division step of dividing the conference into a plurality of sections;
A first selection step of selecting, from the text recorded in the first recording step, text generated from the start time to the end time of the section;
A second selection step of selecting image data generated from the start time to the end time of the section from the image data recorded in the second recording step;
A minutes comprising: a creation step of creating a layout by associating the text selected in the first selection step with the image data selected in the second selection step Generation method.   A program for causing a computer to execute the minutes generation method according to claim 14.