JP2014204411A - Conference recording system, conference recording device, conference recording-reproduction method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a conference recording system for selectively reproducing a content which gathered the attention of participants in a conference.SOLUTION: There is provided a conference recording system comprising a conference recording device 101 for recording a conference content and an information processing device 121 for reproducing a recorded conference content. The conference recording device 101 acquires conference data pertaining to video/audio at a conference, creates, on the basis of a conference data, noteworthy information indicating information to which conference participants paid attention, and records the conference data in correlation with the noteworthy information. The information processing device 121 acquires the conference data from the conference recording device 101 and reproduces the conference data according to the noteworthy information corresponding to the conference data.

Description

  The present invention relates to a conference recording system, a conference recording device, a conference recording / reproducing method, and a computer program.

  There has been proposed a conference recording system that supports video and audio data recorded using a video camera or the like and played back in a timeline format to look back over the entire content of the conference in time series.

  Patent Document 1 discloses a system that detects a speaker during a meeting and a visual recognition area of the speaker, and displays an image of the region that was visually recognized together with a moving image.

JP 2005-124160 A

  It is desirable for the meeting recording system to be able to look back on the contents of the entire meeting in chronological order, and to be able to look back on the contents that attracted the attention of participants during the meeting efficiently and in a short time. ing.

  For example, in an idea presentation meeting where brainstorming or the like is performed, there is a case where it is not necessary to look back on all the statements, and there is a case where it is desired to selectively look back on the ideas that attracted the attention of participants. In such a case, in the related art, it is necessary to browse the moving image for the entire time, remember the idea speaker, etc., and repeat search and reproduction by the speaker name to find the desired content. Therefore, in the prior art, it takes time to look back on the content that attracted attention at the conference.

  Recently, an omnidirectional video camera may be used to record a conference. In the omnidirectional video camera, the output moving image is subjected to perspective projection conversion processing and recorded as a panoramic moving image. Therefore, when the panorama operation recorded by the conference recording system using the omnidirectional video camera is reproduced, the range to be confirmed when looking back on the conference content becomes very wide, so it takes time to find the content that attracted attention.

  The present invention has been made to solve at least one of the above problems. It is an object of the present invention to provide a mechanism for playing back content that attracts the attention of participants in a conference.

  A conference recording system according to an embodiment of the present invention is a system including a conference recording device that records conference content and an information processing device that reproduces the recorded conference content. The conference recording device includes an acquisition unit that acquires conference data related to video / audio in a conference, and a generation unit that generates attention information indicating information that a conference participant has focused on based on the acquired conference data; Recording means for recording the conference data in association with the attention information. The information processing apparatus includes an acquisition unit that acquires the recorded conference data from the conference recording device, and a reproduction unit that reproduces the acquired conference data according to attention information corresponding to the conference data.

  According to the conference recording system of the present invention, it is possible to play back the content focused on the attention of the participants in the conference.

It is a figure which shows the structural example of the meeting recording system of this embodiment. It is a figure which shows the structural example of a meeting recording device. It is a figure explaining acquisition of meeting data. It is a flowchart explaining the example of the operation | movement process of a meeting recording device. It is a figure explaining the example of the calculation process of an attention area. It is a figure which shows the structural example of a server. It is a figure which shows an example of the operation | movement process of a server. It is a figure which shows the structural example of meeting recording data. It is a figure which shows the structural example of information processing apparatus. It is a figure explaining the acquisition process and provision process of meeting record data. It is a flowchart explaining the example of the reproduction | regeneration processing of meeting recording data. It is an example of a screen structure in connection with reproduction | regeneration of meeting recording data. It is a figure explaining the example of an object extraction process. It is a figure explaining the detection of a slide and a person from a panoramic image. It is a figure which shows an example of a structure of attention information. It is a flowchart explaining the reproduction | regeneration processing of meeting recording data. FIG. 11 illustrates a screen display example of the information processing apparatus.

  FIG. 1 is a diagram illustrating a configuration example of a conference recording system according to the present embodiment. The conference recording system shown in FIG. 1 includes a conference recording device 101, a server 131, and an information processing device 121. The conference recording / reproducing method of the present embodiment is realized by the conference recording system shown in FIG. The conference recording apparatus 101, the server 131, and the information processing apparatus 121 communicate via the network 141. In this example, the network 141 is a LAN (Local Area Network).

  The conference recording device 101 records the conference. Specifically, the conference recording apparatus 101 acquires video / audio by capturing a conference using a video camera, and records data regarding the video / audio as conference data. Of course, the conference recording device 101 may receive and record conference data from an external device.

  The conference recording device 101 records conferences held in, for example, an office or a predetermined venue, but the conference to which the present invention is applied may be a group involving visual recognition / speaking behaviors of a plurality of persons. It is not limited to conferences held at predetermined venues. For example, interviews and surveys are also applicable to the present invention.

  The server 131 acquires the conference data recorded by the conference recording device 101 from the conference recording device 101 and accumulates it. The information processing apparatus 121 acquires the stored conference data from the server 131 and reproduces it. The information processing apparatus 121 is a mobile terminal such as a smartphone or a personal computer.

  Each of the conference recording apparatus 101, the server 131, and the information processing apparatus 121 has standard components (for example, a CPU, a memory, a hard disk, and a network I / F) that are mounted on a general-purpose computer. CPU is an abbreviation for Central Processing Unit. I / F is an abbreviation for Interface.

  FIG. 2 is a diagram illustrating a configuration example of the conference recording apparatus. The conference recording apparatus 101 includes a CPU 103, a memory 104, a video camera 105, a moving image / audio processing unit 106, a visual recognition area extraction unit 107, an attention information processing unit 108, an HDD 109, a communication processing unit 110, and a network I / F 111. HDD is an abbreviation for Hard Disk Drive. The processing units included in the conference recording apparatus 101 are connected to each other via a system bus 102.

  The CPU 103 controls the entire conference recording apparatus 101. The memory 104 stores a control program (computer program) for the conference recording apparatus 101. The CPU 103 executes this control program. The video camera 105 records the behavior of participants in the conference.

  FIG. 3A shows a conference in this embodiment. Conference participants 211 to 215 participate in this conference. A conference recording device 101 is installed on a desk in the conference room. In this embodiment, an omnidirectional video camera is used as the video camera 105.

The video / audio processing unit 106 instructs the visual recognition region extraction unit 107 to extract the visual recognition region of the participant from the video included in the conference data (moving image / audio data) acquired by photographing with the video camera 105. The visual recognition area extraction unit 107 extracts the visual recognition area of the participant in accordance with the instruction received from the video / audio processing unit 106. The visual recognition area extraction unit 107 extracts the visual recognition area by performing prediction based on, for example, the pupil position and the face orientation of the participant. As a method for extracting a visual recognition area, the following reference discloses a technique for estimating the line of sight of a subject using an omnidirectional camera and outputting the visual recognition area.
Reference: JP 2012-124767 A

  FIG. 3B shows an omnidirectional screen acquired by photographing with an omnidirectional video camera. The moving image / audio processor 106 performs perspective projection conversion on the omnidirectional image 210 to generate a panoramic image 230 shown in FIG. A reference numeral 224 in FIGS. 3A and 3B indicates a visual recognition area of the conference participant 214. Reference numeral 221 indicates a visual recognition area of the conference participant 211. Reference numeral 222 denotes a visual recognition area of the conference participant 212. Reference numeral 225 indicates a visual recognition area of the conference participant 215.

  The attention information processing unit 108 detects an overlap of the viewing areas of the conference participants. The code | symbol 240 in FIG.3 (C) shows the area | region (overlapping area | region) with which the visual recognition area overlapped. The overlapping area 240 has a two-dimensional coordinate value (x, y) of the upper left origin and the width and height (w, h) of the area, and can specify the location and calculate the size.

  The attention information processing unit 108 generates attention information based on the detection result of the overlapping area. The attention information is information regarding a scene that attracts the attention of the conference participants. Specifically, the attention information includes at least an attention area and a time (attention time) at which attention is collected. The attention area is an area that attracts the attention of the conference participants. The attention area is obtained by ORing the viewing areas between the consecutive frames when the overlapping of the frames in which the overlapping of the viewing areas is detected continues in the conference data. The attention time corresponds to the number of consecutive frames in which the overlapping of the viewing areas is detected in the conference data.

  The attention information processing unit 108 stores the tag set ID indicating the attention information in the HDD 109 in association with the conference data. The stored data is hereinafter referred to as meeting record data. The communication processing unit 110 transmits conference record data to the server 131 via the network I / F 111.

FIG. 4 is a flowchart for explaining an example of operation processing of the conference recording apparatus.
First, the conference recording apparatus 101 receives a process start instruction and sets conference attribute information (step S301). Specifically, the CPU 103 saves the meeting date and time, location, and meeting participants in the meeting record data.

  Next, the moving image / sound processor 106 starts the conference data shooting process by the video camera 105. The visual recognition area extraction unit 107 extracts the visual recognition area of each participant for each frame of the moving image included in the conference data (step S302). Subsequently, the moving image / sound processor 106 performs perspective projection conversion on the omnidirectional image 210 (FIG. 3B) to generate a panoramic image 230 shown in FIG.

  Next, the attention information processing unit 108 performs a hit determination of the viewing area of each conference participant in the current frame of the panoramic image 230 (step S304). The visual recognition area hit determination is to determine whether the visual recognition areas of the conference participants in the panoramic image 230 overlap each other.

  As shown in FIG. 3A, a case where a conference with five conference participants (211 to 215) is recorded will be described as an example. The hit determination in step S304 is performed to determine whether or not more than a prescribed percentage of all the participants are viewing the same part. Assuming that the prescribed ratio is 60%, in the hit determination, it is determined whether three people, which are 60% of all five participants, are viewing the same part at the same time. Note that the prescribed ratio can be changed.

  FIG. 5A is a diagram for explaining the overlapping of the viewing areas in the Nth frame. Assume that three conference participants are looking at the same location. The viewing areas of the three conference participants are three viewing areas 221, 222, and 225 indicated by rectangles. The overlapping area 401 is an area where the three viewing areas overlap.

  In the hit determination in step S304 of FIG. 4, the attention information processing unit 108 determines whether or not the overlapping area 401 is equal to or larger than a specified size (X pixel). Since the visual recognition areas 221 to 225 have a two-dimensional coordinate value (x, y) of the upper left origin and the width and height (w, h) of the area, the coordinate values (x, y) and the width and height (w, h) of the region can be calculated.

  When the overlapping area 401 is less than the specified size, the attention information processing unit 108 determines that the viewing areas of the conference participants do not overlap each other, and the process proceeds to step S312. When the overlapping area 401 is equal to or larger than the specified size, the attention information processing unit 108 determines that the viewing areas of the conference participants overlap each other, and the process proceeds to step S305.

  Next, the attention information processing unit 108 determines whether the flag is ON by the hit determination for the previous frame (step S305). This flag indicates whether the viewing areas of the conference participants overlap each other in the frame. The flag being ON indicates that the viewing areas of the conference participants overlap each other. The flag being OFF indicates that the viewing areas of the conference participants do not overlap each other.

  If the flag is not ON, the process proceeds to step S306. Then, the attention information processing unit 108 sets the flag to ON (step S306), and the process proceeds to step S313. If the flag is ON, the process proceeds to step S307. Then, the attention information processing unit 108 performs a hit determination of the visual recognition area between consecutive frames (step S307). Specifically, it is determined whether the overlapping area in the previous frame (i-1th frame) and the overlapping area in the current frame (i-th frame) overlap each other.

  FIG. 5B is a diagram for explaining the hit determination in step S307. A region 401 is an overlapping region in which the viewing region overlaps with the previous frame (i−1th frame). A region 402 is an overlapping region where the viewing region overlaps the current frame (i-th frame). The region 411 is an overlapping region where the region 401 and the region 402 overlap. The attention information processing unit 108 calculates the size of the region 411. When the calculated size is equal to or larger than the specified size (Y pixel), it is determined to be true in the hit determination of the viewing area between consecutive frames. The attention information processing unit 108 adds 1 to the counter value (increment), and proceeds to step S309. The counter value represents the number of frames that are continuously determined to be true in the hit determination in step S307.

  Next, the attention information processing unit 108 determines whether the counter value is M or more (step S309). The value of M is predetermined. If the counter value is less than M, the process proceeds to step S312. Then, the attention information processing unit 108 turns off the flag and sets the counter value to 0 (step S312). Then, the process proceeds to step S313.

  If the counter value is greater than or equal to M, the process proceeds to step S310. Then, the attention information processing unit 108 associates the tag set ID with the conference data (moving image) and records it as conference recording data (step S310). The tag set ID indicates attention information. The attention information includes the coordinate information of the attention area described above. The attention area is a total area of the viewing areas of the conference participants in the continuous frames that are true in the determination processing in steps S307 and S309. The attention area is obtained by a logical sum operation of the visual recognition areas of the conference participants.

  FIG. 5C is a diagram illustrating an example of a region of interest. In this example, it is assumed that the counter value is M, and the overlapping areas of adjacent frames are overlapped M times consecutively. The shaded portion in the figure is an overlapping area between the (k−1) th and kth frames, which is the same as the area 411 in FIG. 5 (B). A portion surrounded by a dotted rectangle is a total area of the viewing areas having an overlapping area between the (k−1) th and kth frames. That is, the area of the portion surrounded by the dotted rectangle is the attention area.

  The attention information processing unit 108 sets and records the coordinate information of the attention area as the attention information. Specifically, the coordinates (x, y), width (h), and height (w) of the total area are set in an area 513 for conference recording data, which will be described later with reference to FIG. The attention information processing unit 108 sets the number of frames from the determination in S307 of FIG.

  Returning to FIG. 4, the attention information processing unit 108 sets an evaluation value for attention information (step S <b> 311). The evaluation value indicates the attention level of the attention area. For example, it is assumed that continuous M frames are true in the determination processing in steps S307 and S309. The attention information processing unit 108 calculates the evaluation value by, for example, multiplying the size of the overlapping region in each frame included in the M frame by M. The method for calculating the evaluation value is not limited to the one shown here. The attention information processing unit 108 sets an evaluation value to the evaluation value 515 of attention information shown in FIG.

  Next, the CPU 103 determines whether an instruction to end conference recording has been received (step S313). If the conference recording end instruction has not been received, the process proceeds to step S314. Then, the CPU 103 sets the next frame as a processing target (step S314), and returns to step S302. If an instruction to end conference recording is received, the process proceeds to step S315. Then, the CPU 103 transmits the conference record data to the server 131 via the communication processing unit 110 (step S315).

  Since it demonstrated with reference to FIG. 4, the attention information processing part 108 judges whether the time which a visual recognition area overlaps continued for more than regulation time (time equivalent to M frame) (S304, S305, S307). For this purpose, the attention information processing unit 108 determines whether or not the overlapping areas overlap between adjacent frames for a predetermined number of frames or more. Then, when the overlapping area overlaps between adjacent frames for a predetermined number of frames or more, the attention information processing unit 108 determines that the time for which the viewing area overlaps has continued for a predetermined time or more. Then, the attention information processing unit 108 generates information (image coordinate information, time, evaluation value) on the video corresponding to the viewing area for the time that the viewing areas overlap as attention information.

  FIG. 6 is a diagram illustrating a configuration example of a server. The server 131 includes a CPU 133, a memory 134, an HDD 135, a table processing unit 136, a network I / F 137, and a communication processing unit 138. The processing units included in the server 131 are connected to each other via a system bus 132.

  The CPU 133 controls the entire server 131. The memory 134 stores a control program (computer program) for the server 131. The CPU 103 executes this control program. The table processing unit 136 reads and writes to and from the conference record management table stored in the HDD 135. In this example, the table processing unit 136 stores and manages conference record data received from the conference recording apparatus 101 in a conference record management table. The communication processing unit 138 is connected to the LAN 141 via the network I / F 137, performs data transmission / reception with the conference recording apparatus 101 and the information processing apparatus 121, and accesses data to the conference record management table via the table processing unit 136. I do.

  In this embodiment, HTTP (Hyper Text Transfer Protocol) is used as the communication protocol. A general data format such as JSON is used as the data format. JSON is an abbreviation for JavaScript (registered trademark) Object Notation.

  FIG. 7 is a diagram illustrating an example of server operation processing. First, the communication processing unit 138 of the server 131 receives conference record data from the conference recording device 101 connected to the LAN 141 (step S701). This meeting record data is transmitted by the meeting recording apparatus 101 in step S315 of FIG. Then, the table processing unit 136 stores the received conference record data in the conference record management table in the HDD 135 (step S702).

  FIG. 8 is a diagram illustrating a configuration example of conference record data stored in the conference record management table. FIG. 8A shows the structure of conference record data. FIG. 8B shows a configuration of attention information 511. The attention information is associated with the conference record data 501 by the tag set IDs 508 and 512 and stored in the conference record management table.

  The conference record data 501 includes data items such as a conference record data ID 502, a date and time 503, a time 504, a place 505, a participant 506, data 507, and a tag set ID 508.

  The conference record data ID 502 is an identifier for uniquely identifying conference record data. Date and time 503 is the date and time of the meeting. Time 504 is a meeting time. A place 505 is a meeting place. Participant 506 is a conference participant. The conference recording apparatus 101 sets the date, time, place, and participant of the conference in step S301 in FIG.

  Data 507 is video data of the conference recorded by the video camera 105. The conference recording apparatus 101 sets moving image data in step S310 of FIG. The tag set ID 508 is an identifier that uniquely identifies attention information. A plurality of tag set IDs can be set for one conference record data.

  The attention information 511 includes data items such as a tag set ID 512, an area 513, a time 514, and an evaluation value 515. The tag set ID 512 corresponds to the tag set ID 508 of the conference recording data 501. A region 513 is a region of interest. In this example, the attention area has coordinate information of the coordinate value (x, y) of the upper left origin and the width / height (w, h) of the area. The conference recording apparatus 101 sets the area 513 in step S310 of FIG. Time 514 is time information corresponding to the time of attention information, that is, the number of consecutive frames determined to be true in the hit determination of the visual recognition area. The conference recording apparatus 101 sets time 514 in step S310 in FIG. The evaluation value 515 is an evaluation value of attention information. The conference recording apparatus 101 sets an evaluation value 515 in step S311 of FIG.

  FIG. 9 is a diagram illustrating a configuration example of the information processing apparatus. The information processing apparatus 121 includes a CPU 123, a memory 124, an HDD 125, a reproduction processing unit 126, a communication processing unit 127, a network I / F 128, a display 129, and a display processing unit 130. The processing units included in the information processing apparatus 121 are connected to each other via a system bus 122.

  The CPU 123 controls the entire information processing apparatus 121. The memory 124 stores a control program (computer program) for the information processing apparatus 121. The CPU 123 executes this control program.

  The communication processing unit 127 is connected to the LAN 141 via the network I / F 128 and receives conference record data from the server 131. The communication processing unit 127 stores the received conference record data in the HDD 125. The reproduction processing unit 126 reads the conference record data stored in the HDD 125. Further, the reproduction processing unit 126 performs reproduction processing according to attention information indicated by the tag set ID included in the conference recording data. Specifically, the reproduction processing unit 126 reproduces the moving image of the conference recording data on the display 129 via the display processing unit 130.

  FIG. 10 is a diagram for explaining conference record data acquisition processing and conference record data provision processing. FIG. 10A shows conference record data acquisition processing.

  First, the communication processing unit 127 of the information processing device 121 transmits a conference record data acquisition request to the server 131 (step S1001). The meeting record data acquisition request includes user account information and time information. For example, when the user account information is that of the conference participant 211, the communication processing unit 127 makes a request for acquisition of conference record data of the conference in which the conference participant 211 has participated. The input of account information will be described later with reference to FIG.

  The time information is used by the information processing apparatus 121 to request acquisition of unrecorded conference record information. The time information last requested by the information processing apparatus 121 is stored in the memory 124, the acquisition request is made using the time information, and the previous acquisition request is made without receiving the previously received conference record data. Request acquisition of only meeting record data newer than the specified time. The server processing for this acquisition request will be described later with reference to FIG.

  Next, the communication processing unit 127 receives the conference record data corresponding to the acquisition request from the server 131 and stores it in the HDD 125 (step S1002). Then, the display processing unit 130 displays the conference record data (step S1003). The display screen of the conference record data will be described later with reference to FIG.

  FIG. 10B shows conference record data provision processing executed by the server in response to an acquisition request from the information processing apparatus.

  First, the communication processing unit 138 of the server 131 receives a conference record data acquisition request from the information processing apparatus 121 (step S1011). The table processing unit 136 uses the account information and the time information included in the acquisition request to search for conference record data corresponding to the condition from the conference record management table (step S1012).

  Next, the table processing unit 136 determines whether there is meeting record data corresponding to the meeting record management table 501 based on the search result of the meeting record data (step S1013).

  If there is meeting record data corresponding to the meeting record management table 501, the process proceeds to step S1014. If there is no meeting record data corresponding to the meeting record management table 501, the process proceeds to step S1015.

  In step S <b> 1014, the communication processing unit 138 transmits the conference record data obtained by the search to the information processing apparatus 121. In step S1013, the communication processing unit 138 notifies the information processing apparatus 121 that the conference record data corresponding to the acquisition request (S1001 in FIG. 10A) is not stored.

  FIG. 11 is a flowchart illustrating an example of a process for reproducing conference record data by the information processing apparatus. FIG. 12 is a screen configuration example related to the reproduction of conference record data in the information processing apparatus. As shown in FIG. 12A, a dedicated application is used to reproduce the conference record data in the information processing apparatus 121.

  When the dedicated application is activated in response to a user instruction, the information processing apparatus 121 displays the screen illustrated in FIG. 12A on the display 129 and accepts input of an account name and a password (step S1101). The information processing apparatus 121 executes an authentication process using the account name and password that have been accepted. This authentication process is necessary to use the conference record data.

  Next, the information processing apparatus 121 determines whether the authentication is successful (step S1102). If the authentication fails, the information processing apparatus 121 does not perform subsequent display processing. If the authentication is successful, the playback processing unit 126 displays a list of conference recording data stored in the HDD 125 on the display 129 via the display processing unit 130 (step S1103). The reproduction processing unit 126 may preferentially display the conference record data having a high evaluation value 515 (FIG. 8B) in the attention information associated with the conference record data at the top of the list.

  FIG. 12B shows a display screen for a list of conference record data. In this example, the list of conference record data displays information that can uniquely identify the conference in which the conference participant has participated, such as the conference location, the participant name, and the date and time.

  Returning to FIG. 11, the user of the information processing apparatus 121 selects conference record data to be reproduced from the list of conference record data. As a result, the information processing apparatus 121 receives an instruction to select the conference recording data (step S1104).

  The reproduction processing unit 126 displays the conference record data on the display 129 in accordance with the accepted selection instruction (step S1105).

  FIG. 12C shows a screen displayed in step S1105 of FIG. The reproduction processing unit 126 displays the meeting location, participants, time, and the like as attribute information of the meeting in the display field 621. In addition, the reproduction processing unit 126 displays the moving image included in the conference record data in the display field 622. The user can issue a playback instruction for the moving image by touching the display field 622 or the like.

  Next, in step 1106, the reproduction processing unit 126 determines whether or not an instruction to reproduce the moving image displayed in the display field 622 in FIG. When the reproduction processing unit 126 receives a moving image reproduction instruction, the process proceeds to step S1107. If the reproduction processing unit 126 has not received a moving image reproduction instruction, the process returns to step S1106.

  In step 1107, the reproduction processing unit 126 reproduces the moving image according to the attention information associated with the conference record data (step S1107). Specifically, the reproduction processing unit 126 acquires attention information 511 associated with tag set IDs 508 and 512 (FIGS. 8A and 8B) corresponding to the moving image for which the reproduction instruction has been received. Then, the reproduction processing unit 126 reproduces the moving image for the time set in the time 514 with respect to the attention area 513 included in the acquired attention information.

  The conference recording system of the present embodiment extracts the visual recognition area of each participant when recording the conference, detects that the multiple visual recognition areas overlap, and if the visual recognition area continues for a certain time or more, the overlapping visual recognition area In addition, the attention information tag is assigned to the moving image portion of time and recorded. As a result, after the conference, it is possible to reproduce the conference contents focused on by the conference participants and review the conference contents in a short time.

(Example 2)
The conference recording system according to the second embodiment adds object information to attention information corresponding to conference recording data, and performs moving image reproduction according to the object information. The object information is type information indicating the type of object included in the attention information. The conference recording system according to the second embodiment is different from the conference recording system according to the first embodiment in that when recording conference recording data, object information is added and recorded, and the playback method is switched according to the object indicated by the type information. Only thing is different. Therefore, the same parts as those in the first embodiment are designated by the same reference numerals and omitted, and only different parts will be described below.

FIG. 13 is a diagram for explaining an example of object extraction processing executed by the conference recording apparatus.
In this example, the object extraction process is executed between step S310 and step S311 in FIG. In this embodiment, the objects to be extracted are a slide, a person, and others. A slide refers to a slide on which data is projected by a projector or the like.

  FIG. 15 is a diagram illustrating an example of a configuration of attention information in the second embodiment. The attention information 911 includes an object type 916 in addition to the tag set ID 512 to the evaluation value 515. The object type 916 is object information.

  Returning to FIG. 13, the attention information processing unit 108 of the conference recording apparatus 101 determines whether or not the attention information associated with the conference recording data recorded in the process of step S310 in FIG. S1301). Specifically, the attention information processing unit 108 determines whether the attention area included in the attention information includes a slide identifier. The slide identifier is identification information for identifying a slide range (slide area). The slide identifier is, for example, a marker embedded in the slide.

  If there is no slide identifier in the attention information, the process proceeds to step S1305. If the attention information includes a slide identifier, the process proceeds to step S1302. Then, the attention information processing unit 108 extracts a slide area from the panoramic image 230 that is the original moving image of the attention information based on the slide identifier (step S1302).

  FIG. 14A is a diagram for explaining the extraction process of the slide area from the panoramic image. An area 801 surrounded by a thick line frame in the panoramic image 230 is an attention area included in the attention information. When the attention area 801 includes the slide identifier 803, the attention information processing unit 108 extracts the slide area 802 from the panoramic image 230 using the slide identifier 803.

  Returning to FIG. 13, the attention information processing unit 108 records “slide” in the object type 916 included in the attention information (step S <b> 1303). Subsequently, the attention information processing unit 108 updates and records the coordinate information of the attention area 513 included in the attention information with the coordinate information of the extracted slide area (step S1304).

  Next, the attention information processing unit 108 updates and records the time 514 included in the attention information to a time corresponding to one frame (step S1305). This is because when the object included in the attention information is a slide, the attention information can be reproduced as a still image instead of a moving image.

  Next, the attention information processing unit 108 performs a moving object detection process on the attention information (step S1306). The attention information processing unit 108 determines whether there is a person in the attention area, that is, whether the object in the attention area is a person based on the result of the moving object detection process (step S1307). If there is no person in the attention area, the process proceeds to step S1312. If there is a person in the attention area, the process proceeds to step S1308. The attention information processing unit 108 records “person” in the object type 916 included in the attention information.

  FIG. 14B is a diagram illustrating detection of a person from a panoramic image. When the attention information processing unit 108 detects that a person is in the attention area 804 in the panoramic image 230, the attention information processing unit 108 records “person” in the object type 916.

  Returning to FIG. 13, the attention information processing unit 108 instructs the video / audio processing unit 106 to analyze the conference recording data, and extracts a speech section including the time 514 of the attention information (step S 1309).

  Next, the attention information processing unit 108 determines whether the utterance section is longer than the time 514 of the attention information (step S1310). If the utterance section is longer than the time 514 of the attention information, the process proceeds to step S1311. Then, the attention information processing unit 108 overwrites and records the time 514 of the attention information with the utterance section (step S1311). This is because when there is a person in the region of interest, there is a high possibility that the user is speaking, and the utterance section is divided halfway to prevent content understanding from being hindered. If it is determined in step S1310 that the utterance section is shorter than the attention information time, the attention information time 514 is not updated.

  In step S1312, the attention information processing unit 108 determines whether the slide identifier is extracted in the determination process of step S1301 (step S1312). If the slide identifier is extracted, the process ends. If the slide identifier has not been extracted, the process proceeds to step S1313. Then, the attention information processing unit 108 records “others” in the object type 916 without changing the area 513 and the time 514 included in the attention information.

  The processing described with reference to FIGS. 13 to 15 makes it possible to record what appears in the attention information associated with the conference record data.

  FIG. 16 is a flowchart for explaining the process of reproducing conference record data in the second embodiment. The information processing apparatus 121 acquires the conference record data from the server 111 and switches the method for reproducing the conference record data according to the object type 916. First, the reproduction processing unit 126 of the information processing apparatus 121 displays a list of conference recording data (step S1501). For example, the reproduction processing unit 126 displays the screen shown in FIG.

  Next, it is judged whether the reproduction | regeneration processing part 126 received the selection instruction | indication of meeting recording data (step S1502). If a meeting record data selection instruction has not been received, the process returns to step S1502. If a meeting record data selection instruction has been received, the process advances to step S1503.

  Next, the reproduction processing unit 126 determines the object type 916 of attention information associated with the selected conference record data (step S1503). If “slide” is designated as the object type 916, the process advances to step S1504. Then, the reproduction processing unit 126 displays the slide image (still image reproduction) according to the area information 913 included in the attention information (step S1504).

  FIG. 17 is a diagram illustrating a screen display example of the information processing apparatus. The reproduction processing unit 126 displays a slide image in the area 1703.

  Returning to FIG. 16, when “person” or “other” is designated as the object type 916 of the attention information, the reproduction processing unit 126 reproduces the moving image according to the area information 913 and the time 514 included in the attention information. (Step S1505). Specifically, the reproduction processing unit 126 reproduces the moving image for the time set in the time 514 in the area 1704 in FIG. When the object type is a person, the time set for the time 514 is the time after the update by the process of step S1311 in FIG. Therefore, in this case, the reproduction processing unit 126 reproduces the moving image for the time corresponding to the occurrence section set as the time of the attention information.

  According to the conference recording system of the present embodiment, it is possible to perform moving image playback in an appropriate playback time according to the object type corresponding to the area that attracted attention in the conference.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

101 conference recording device 121 information processing device 131 server

Claims (13)

A system comprising a conference recording device that records conference content, and an information processing device that reproduces the recorded conference content,
The conference recording device
Acquisition means for acquiring conference data relating to video / audio at the conference;
Creating means for creating attention information about a scene that a conference participant has paid attention to based on the acquired meeting data;
Recording means for recording the conference data in association with the attention information,
The information processing apparatus includes:
Obtaining means for obtaining the recorded conference data from the conference recording device;
A conference recording system comprising: a reproducing unit that reproduces the acquired conference data in accordance with attention information corresponding to the conference data. The creation means provided in the conference recording device includes:
Extracting the viewing area of the conference participant from the video included in the acquired conference data;
Determining whether the extracted viewing area overlaps for a specified time or more,
The information regarding the image | video corresponding to the said visual recognition area for the time for which the said visual recognition area overlaps is produced | generated as said attention information when the time when the said visual recognition area overlaps more than regulation time. Conference recording system. The creation means provided in the conference recording device includes:
Obtaining an overlapping area where the viewing area of the conference participant overlaps in the video frame included in the conference data;
Determining whether or not the overlapping region overlaps between adjacent frames continues for a predetermined number of frames or more,
3. The conference recording system according to claim 2, wherein when the overlapping area overlaps between adjacent frames for a predetermined number of frames or more, it is determined that the time for which the visual recognition area overlaps continues for a specified time or longer. The creation means provided in the conference recording device includes:
Corresponds to the coordinate information of the region of interest, which is the region obtained by the logical OR operation of the visual recognition region in the continuous frame, and the number of the continuous frames, when the overlapping regions overlap for the specified number of frames or more. 4. The conference recording system according to claim 3, wherein time information to be set is set as the attention information. Based on the coordinate information and the time information of the attention area set in the attention information corresponding to the conference data, the reproduction means included in the information processing apparatus is a moving image corresponding to the time indicated by the time information of the attention area. The conference recording system according to claim 4, wherein the conference recording system is reproduced. The reproduction means provided in the information processing apparatus displays the conference data list on the screen,
6. The conference recording system according to claim 5, wherein a moving image is reproduced for a region of interest set in attention information corresponding to conference data selected from the list. The creation means provided in the conference recording device further includes:
Extracting an object from the attention area set in the attention information;
Set the type information indicating the type of the extracted object in the attention information,
The reproduction means included in the information processing apparatus switches the reproduction method of the conference data according to the type of the object indicated by the type information set in the attention information corresponding to the acquired conference data. The conference recording system according to any one of claims 4 to 6. The creation means provided in the conference recording device includes:
When the object extracted from the attention area set in the attention information is a slide, the coordinate information of the attention area set in the attention information is updated to the coordinate information of the area of the slide, and the attention information is updated. Update the set time information to the time information for one frame,
The conference recording system according to claim 7, wherein the reproduction unit included in the information processing apparatus reproduces the slide as a still image according to the updated time information. The creation means provided in the conference recording device includes:
When the object extracted from the attention area set in the attention information is a person, the occurrence section including the time information set in the attention information is extracted from the meeting data,
When the extracted occurrence interval is longer than the time indicated by the time information set in the attention information, the time information set in the attention information is updated in the occurrence interval,
The conference recording system according to claim 7, wherein the reproduction unit included in the information processing apparatus reproduces the attention area as a moving image for a time indicated by the updated time information. 10. The conference recording according to claim 1, wherein the creation unit included in the conference recording apparatus acquires the conference data obtained by photographing the conference using an omnidirectional camera. system. Acquisition means for acquiring conference data relating to video / audio at the conference;
Creating means for creating attention information indicating information noticed by a conference participant based on the acquired meeting data;
Recording means for recording the conference data in association with the attention information,
The recorded conference data is reproduced according to attention information corresponding to the conference data. A conference recording / reproducing method in a system comprising a conference recording device for recording conference content and an information processing device for reproducing the recorded conference content,
The conference recording device acquiring conference data relating to video / audio in a conference;
The conference recording device, based on the acquired conference data, creating attention information indicating information focused on by the conference participants;
The conference recording device recording the conference data in association with the attention information;
The information processing device acquiring the recorded conference data from the conference recording device;
The information processing apparatus includes a step of reproducing the acquired conference data according to attention information corresponding to the conference data. A computer program for causing a computer to execute the conference recording / reproducing method according to claim 12.

Images