JP7102859B2 - Video Conference Systems, Video Conference Methods, and Programs - Google Patents

Description

The present invention relates to video conference systems, video conference methods, and programs.

A video conference system is used as one of the methods for holding a conference with a person in a remote place. In a video conference system, by exchanging images taken at each location with each other, it is possible to hold a conference between people who are separated from each other.

The technique related to the video conference system described above is disclosed in, for example, Patent Document 1 below. In the following Patent Document 1, (1) each participant is authenticated using an image of a participant in a remote conference, and (2) an action permission process is performed when the number of authenticated participants reaches the required number. A technique for executing (a process of notifying that a meeting can be started) is disclosed.

Further, in Patent Document 2 below, (1) a user existing in front of the display is detected by using images of a plurality of image pickup devices provided on the display, and (2) a display is displayed based on the detection result of the user. Disclosed is a technique for arranging a display in a position that is easy for all users to see by moving the position of the display in the vertical and horizontal directions.

Japanese Unexamined Patent Publication No. 2009-171119 Japanese Unexamined Patent Publication No. 2016-004207

In the above-mentioned Patent Document 1, each person is specified (authenticated) by using an image showing a person participating in the conference. Here, in most cases, the participants in the conference have different orientations or postures with respect to the provided cameras. Therefore, it is possible that at least some of the participants in the conference cannot be identified from the image. Also, given the nature of the conference, it is not desirable for the participants in the conference to remain unidentified as individuals.

The present invention has been made in view of the above problems. One of the objects of the present invention is to provide a technique for identifying a participant of a conference in a video conference system.

The video conference system of the present invention
An image acquisition means for acquiring an image of a person participating in the conference generated by the first imaging device, and
A person identification means that analyzes the image and executes a person identification process for identifying a person included in the image, and
A position detecting means for detecting the position of an unspecified person who could not be identified by the person identification process, and
A process execution means for executing a predetermined process for identifying the unspecified person using the detected position of the unspecified person, and a process executing means.
To be equipped.

The video conference method of the present invention
The computer
Acquire the image of the participants of the conference generated by the first imaging device,
The image is analyzed, a person identification process for identifying the person included in the image is executed, and the person identification process is executed.
Detects the position of an unspecified person that could not be specified by the person identification process,
Using the detected position of the unspecified person, a predetermined process for identifying the unspecified person is executed.
Including that.

The program of the present invention causes a computer to perform the video conferencing method described above.

According to the present invention, in a video conference system, it is possible to identify the participants in a conference.

It is a figure which shows the configuration example of the video conference system in 1st Embodiment. It is a block diagram which illustrates the hardware configuration of a video conference system. It is a flowchart which illustrates the flow of the process executed by the video conference system of 1st Embodiment. It is a flowchart which illustrates the flow of the process executed by the video conference system of 1st Embodiment. It is a figure which shows the configuration example of the video conference system of 2nd Embodiment. It is a figure which shows the configuration example of the video conference system of 3rd Embodiment. It is a figure which shows an example of the information for identifying a subject, which is displayed on a display device. It is a figure which shows the configuration example of the video conference system in 4th Embodiment. It is a flowchart which illustrates the flow of the process executed by the video conference system of 4th Embodiment. It is a figure which shows the configuration example of the video conference system in 5th Embodiment. It is a figure which illustrates the hardware structure of the video conference system in 5th Embodiment. It is a flowchart which illustrates the flow of the process executed by the video conference system of 5th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings, similar components are designated by the same reference numerals, and description thereof will be omitted as appropriate. Further, unless otherwise specified, in each block diagram, each block represents a configuration of a functional unit, not a configuration of a hardware unit.

[First Embodiment]
[System configuration example]
FIG. 1 is a diagram showing a configuration example of the video conference system 1 according to the first embodiment. In the video conference system 1 illustrated in FIG. 1, the server device 10 and a plurality of communication terminals 20 provided at each point where the conference is held are connected to each other so as to be able to communicate with each other. The video conference is executed between the communication terminals 20 via the server device 10.

An imaging device 30 and a display device 40 are connected to the communication terminal 20. The image pickup device 30 captures an image M of a person who participates in the conference at a point where the image pickup device 30 is provided and displays it on a display device 40 provided at a point on the other side of the video conference (hereinafter, "main"). Notation "image") is generated. The display device 40 displays the main image M of the other party, which is taken by the image pickup device 30 provided at the other party's point of the video conference. Further, information (for example, name, affiliation, etc.) of a participant specified by using the main image M is superimposed and displayed on the main image M displayed on the display device 40 on the other side. Further, in the video conference system 1, a mobile image pickup device 35 is provided in addition to the image pickup device 30. As an example, the imaging device 35 is incorporated in a robot that can move autonomously. As another example, the image pickup device 35 is incorporated in a portable terminal such as a smartphone, a tablet, or a notebook personal computer. When there is a person who could not be identified by using the main image M generated by the image pickup device 30, the image pickup apparatus 35 uses an image S (hereinafter, referred to as “sub-image”) to identify the person. Generate.

As shown in FIG. 1, the video conference system 1 includes an image acquisition unit 110, a person identification unit 120, a position detection unit 130, and a processing execution unit 140. In the example of FIG. 1, these processing units are provided in one server device 10, but the configuration of the video conference system 1 is not limited to the example of FIG. Although not shown, all or part of these processing units may be distributed or overlapped in a plurality of server devices.

The image acquisition unit 110 acquires an image of a person participating in the conference generated by the image pickup device 30 (first image pickup device). In the example of FIG. 1, the image acquisition unit 110 can acquire an image of a person participating in the conference generated by the image pickup apparatus 30 from a communication terminal 20 connected via a network. Further, the image acquisition unit 110 obtains an image of a person participating in the conference from an image pickup device (first image pickup device) connected to another communication terminal (not shown) at a place where the other communication terminal is provided. Can be obtained.

The person identification unit 120 analyzes the image acquired by the image acquisition unit 110 and executes a person identification process for identifying a person included in the image. In other words, the person identification unit 120 individually identifies (authenticates) the participants appearing in the image acquired by the image acquisition unit 110.

The person identification unit 120 operates as follows, for example. First, the person identification unit 120 detects an area recognized as a person from the image acquired by the image acquisition unit 110. The person identification unit 120 can detect a region recognized (classified) as a "person" by using a known general object detection algorithm. Further, the person identification unit 120 may detect, for example, a region of a moving object as a region of a person. The person identification unit 120 can determine the "moving object" based on, for example, the amount of movement of feature points between a plurality of images arranged in time series. Specifically, the person identification unit 120 can estimate the area of the object including the feature points moving by the reference value or more between the plurality of images arranged in the time series as the area of the person. Then, the person identification unit 120 determines who is a person included in each area based on the result of collating the feature amount extracted from the detected area with the feature amount of the participant registered in advance. Can be identified. The feature amount of the participants in the conference is stored in advance in the storage device of the server device 10 or the like in association with the information (name, affiliation, etc.) of the participants. Here, when the person included in the detected area cannot be identified, the person identification unit 120 associates the information indicating that the person included in the area cannot be identified (specific failure information) with the area. The "case where a person cannot be identified" is, for example, a case where there is no person whose score calculated as a result of collation is equal to or higher than the reference value.

The position detection unit 130 detects the position of the unspecified person when there is a person (hereinafter referred to as “unspecified person”) that could not be specified by the person identification process executed by the person identification unit 120. In the example of FIG. 1, as a result of the person identification process using the main image M acquired by the image acquisition unit 110, three of the four participants (Jane, John, Nancy) were identified. In, the case where the remaining one person cannot be identified and is determined to be an unspecified person is depicted. In this case, the position detection unit 130 detects the position of the remaining one unspecified person who could not be specified. As an example, the position detection unit 130 can detect the coordinates (position) of an unspecified person in the coordinate system of the main image M as it is. As another example, the position detection unit 130 uses the coordinates (position) of an unspecified person in the coordinate system of the main image M as the coordinate in another coordinate system (for example, the coordinates in the map data of the place where the conference is held). The coordinates (position) of an unspecified person in the system) may be detected. In this case, a rule (conversion parameter) for converting the coordinates in the coordinate system of the main image M to the coordinates in another coordinate system is prepared in advance in, for example, the memory or the storage device of the server device 10. The rules (conversion parameters) for converting the coordinates in the coordinate system of the main image M to the coordinates in another coordinate system are, for example, the imaging range of the imaging device 30 provided at the place where the meeting is held and the seat position at that place. It can be generated based on the map data including. Information for generating conversion rules (conversion parameters) is transmitted to the server device 10 from a terminal (not shown) used by the participants at each point before the conference is held, for example. As an example, the server device 10 displays the map data held by the robot 60 and the image generated by the image pickup device 30 on the display device 40 side by side or can be switched, and the position of each seat in the image and the map data. Accepts input that associates with the seat position of. Based on this input, the server device 10 can generate a rule (conversion parameter) for converting the coordinate system of the image into the coordinate system of the map data.

The processing execution unit 140 executes a predetermined process for identifying the unspecified person by using the position of the unspecified person detected by the position detecting unit 130. Details of the predetermined processing executed by the processing execution unit 140 will be described later. By the predetermined processing of the processing execution unit 140, the image acquisition unit 110 can acquire the sub-image S by the image pickup apparatus 35. Then, the person identification unit 120 executes a person identification process using the sub-image S to identify an unspecified person.

[Hardware configuration example]
Each functional component of the video conference system 1 may be realized by hardware that realizes each functional component (eg, a hard-wired electronic circuit, etc.), or a combination of hardware and software (eg, example). It may be realized by a combination of an electronic circuit and a program that controls it). Hereinafter, a case where each functional component of the video conference system 1 is realized by a combination of hardware and software in the server device 10 will be further described.

FIG. 2 is a block diagram illustrating a hardware configuration of the video conference system 1. In the example of FIG. 2, the server device 10 has a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

The bus 1010 is a data transmission path for the processor 1020, the memory 1030, the storage device 1040, the input / output interface 1050, and the network interface 1060 to transmit and receive data to and from each other. However, the method of connecting the processors 1020 and the like to each other is not limited to the bus connection.

The processor 1020 is a processor realized by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

The storage device 1040 is an auxiliary storage device realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, a ROM (Read Only Memory), or the like. The storage device 1040 stores a program module that realizes each function of the video conference system 1 (image acquisition unit 110, person identification unit 120, position detection unit 130, processing execution unit 140, and the like). When the processor 1020 reads each of these program modules into the memory 1030 and executes them, each function corresponding to the program module is realized.

The input / output interface 1050 is an interface for connecting the server device 10 and various input / output devices. An input device (not shown) such as a keyboard or mouse, or an output device (not shown) such as a display or speaker may be connected to the input / output interface 1050.

The network interface 1060 is an interface for connecting the server device 10 to the network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method of connecting the network interface 1060 to the network may be a wireless connection or a wired connection. As shown, the server device 10 is communicably connected to a plurality of communication terminals 20 provided at each point where a video conference is held via a network interface 1060. Each communication terminal 20 has an image pickup device 30 for photographing a person participating in the meeting, a display device 40 for displaying an image generated by each image pickup device 30, and a sound collecting device for picking up the sound during the meeting. The device 50 is connected. Further, an audio output device (not shown) for outputting the audio of the conference is further connected to each communication terminal 20. Further, the server device 10 is connected to an image pickup device 35 different from the image pickup device 30 (first image pickup device) that generates the main image M via the network interface 1060.

The image acquisition unit 110 can acquire an image of a person participating in the conference from each communication terminal 20 via the network interface 1060. Further, the server device 10 can transmit an image of a participant of the other party to each communication terminal 20 via the network interface 1060. Further, the server device 10 can transmit an image of a participant at a point where the communication terminal 20 is provided to each communication terminal 20 via the network interface 1060.

[Processing flow]
A flow of processing executed by the video conference system 1 of the first embodiment will be described with reference to FIGS. 3 and 4. 3 and 4 are flowcharts illustrating a flow of processing executed by the video conference system 1 of the first embodiment.

First, the image acquisition unit 110 acquires the main image M in which the participants of the conference at a certain point are captured from the communication terminal 20 (S102). The image acquisition unit 110 can acquire the main image M generated by the image pickup apparatus 30 via the network interface 1060.

The person identification unit 120 detects an area recognized as a person from the main image M acquired in the process of S102 (S104). Then, the person identification unit 120 executes the person identification process for the area detected by the process of S104 (S106). When the person cannot be identified (S108: NO), the person identification unit 120 provides information indicating that the person included in the area cannot be identified with respect to the area targeted for the person identification process (identification failure). Information) is associated (S110). On the other hand, when the person can be identified (S108: YES), the person identification unit 120 acquires the person information including the name of the specified person and associates it with the area of the person (S112). The person information is acquired in association with the feature amount of the participants before the meeting is held, and is registered in advance in the storage device 1040 or the like. In addition to the name of the person, the person information includes the name of the group (company, department, etc.) to which the person belongs, the title of the person, and the like. The processes of S106 to S110 are repeated until all the areas of the person detected in the process of S104 are processed (S114: NO).

After all the areas of the person detected in the process of S104 are processed (S114: YES), the server device 10 transfers the person information of each person associated with the process of S112 to the area to which the person information is associated. It is superimposed on the main image M according to the position of, and transmitted to the communication terminal 20 at another point. As a result, the result of the person identification process using the main image M at a certain point is displayed on the display device 40 at another point together with the main image M (S116).

Further, the position detection unit 130 determines whether or not an unspecified person exists (S118). Here, the position detection unit 130 can determine whether or not an unspecified person exists depending on whether or not there is an area associated with the specific failure information. If there is no unspecified person (S118: NO), the subsequent processing is not executed.

When an unspecified person exists (S118: YES), the position detection unit 130 detects the position of the unspecified person (S120). Here, the position detection unit 130 may detect the coordinates (position) of the unspecified person in the coordinate system of the main image M, or may detect the coordinates (position) of the unspecified person in the coordinate system of the main image M. Based on this, the coordinates (position) of an unspecified person in another coordinate system may be detected.

Then, the process execution unit 140 executes a predetermined process for identifying the unspecified person by using the position of the unspecified person detected by the position detection unit 130 (S122). The unspecified person is identified by using the sub-image S generated by the mobile imaging device 35, which is different from the imaging device 30. From another point of view, the predetermined process executed by the process execution unit 140 can be said to be a process for acquiring a sub-image S in which an unspecified person is captured in a manner different from that of the main image M. As an example, the processing execution unit 140 provides information for specifying a shooting position (a position where the imaging device 35 should execute a shooting operation) to an autonomously movable robot in which a mobile imaging device 35 is incorporated. The process of generating and outputting is executed as a predetermined process. As another example, the process execution unit 140 executes a process of generating and outputting information for identifying a subject (unspecified person) of the image pickup apparatus 35 as a predetermined process.

The image acquisition unit 110 acquires an additional image (sub-image S) generated by the image pickup device 35 in response to a predetermined process of the process execution unit 140 from the device in which the image pickup device 35 is incorporated (S124). Then, the person identification unit 120 identifies an unspecified person by analyzing the sub-image S (S126). Here, the person identification unit 120 acquires the person information of the participant specified by using the sub-image S from the person information registered in advance in the storage device 1040 or the like. Then, the person identification unit 120 associates the acquired person information with the area of the unspecified person in the main image M (S128). The server device 10 superimposes the person information acquired in the process of S128 on the main image M according to the position of the area associated with the person information, and transmits the person information to the communication terminal 20 at another point. As a result, the result of the person identification process using the sub-image S is additionally displayed on the display device 40 at another point (S130).

It should be noted that there is a possibility that a sub-image S in which an unspecified person is unclear may be acquired depending on the environment at the time of shooting. If the unspecified person cannot be identified even after analyzing the sub-image S, the person-identifying unit 120 may output a retake instruction of the sub-image S to the device in which the imaging device 35 is incorporated. Further, when the unspecified person cannot be identified even though the sub-image S is retaken a predetermined number of times, the person identification unit 120 preliminarily sets the unspecified person as an outsider (as a participant in the meeting). It may be determined as a person other than the registered person). In this case, the person identification unit 120 may execute a process of notifying the presence of an outsider by using a display device 40, a speaker (not shown), or the like.

As described above, in the present embodiment, when there is a person who cannot be identified using the main image M, the sub-image S generated by the mobile image pickup device 35, which is different from the image pickup device 30 that generated the main image M, is displayed. The process of identifying the unspecified person is executed by using. As a result, it is possible to prevent an unspecified person from being left unspecified at the meeting place. Further, according to the present embodiment, the participants of the conference can recognize the existence of an outsider who has been mixed into the meeting place.

[Second Embodiment]
In the present embodiment, the processing when the image pickup apparatus 35 is incorporated in the autonomously movable robot will be described. The present embodiment is the same as the first embodiment except for the points described below.

[System configuration example]
FIG. 5 is a diagram showing a configuration example of the video conference system 1 of the second embodiment. As shown in FIG. 5, the image pickup apparatus 35 of the present embodiment is incorporated in the autonomously movable robot 60. Further, in the present embodiment, the processing execution unit 140 generates information for specifying the shooting position of the imaging device 35 based on the position of the unspecified person. Further, the processing execution unit 140 outputs information for specifying the shooting position of the imaging device 35 to the robot 60, and guides the robot 60 to the position specified by the information to execute shooting.

The processing execution unit 140 identifies the shooting position of the image pickup device 35 incorporated in the robot 60 from the position of the unspecified person detected by the position detection unit 130. Here, the processing execution unit 140 calculates the shooting position of the imaging device 35 as a position on the map data held by the robot 60. In the present embodiment, the position detection unit 130 may detect the coordinates (position) of the unspecified person in the coordinate system of the main image M, or the coordinates of the unspecified person in the coordinate system of the main image M. The coordinates (position) of an unspecified person in the coordinate system of the map data may be detected based on (position). In the former case, the processing execution unit 140 uses a rule (conversion parameter) for converting the coordinates (position) of an unspecified person in the coordinate system of the main image M into the coordinates (position) in the coordinate system of the map data. The imaging position of the image pickup device 35 is calculated. Then, the calculated information for specifying the shooting position is output to the robot 60. In the latter case, the processing execution unit 140 may output information for identifying the position detected by the position detection unit 130 to the robot 60. Further, the processing execution unit 140 can generate information indicating an angle at the time of shooting (information indicating how much the image pickup apparatus 35 is tilted in which direction) on the robot 60, and output the information in association with the shooting position. Specifically, the processing execution unit 140 further determines the position (height) of the face of an unspecified person in the main image M, and can image the image pickup device 35 based on the position of the face and the shooting position. The angle at the time of shooting can be calculated based on the range.

The robot 60 moves to a position specified by the information acquired from the processing execution unit 140. The robot 60 can calculate its own position using a known self-position estimation algorithm and move to a target position (a position specified by information acquired from the processing execution unit 140). When the robot 60 reaches a target position, the robot 60 causes the image pickup device 35 to perform a shooting operation. As a result, a sub-image S in which an unspecified person who has not been specified using the main image M is captured is generated. Further, the robot 60 transmits the sub-image S generated by the image pickup apparatus 35 to the image acquisition unit 110. As a result, as described in the first embodiment, the person identification process using the sub-image S can be executed.

Further, a person may be able to operate the movement of the robot 60. For example, a person participating in the conference may operate a portable terminal (smartphone, laptop computer, etc.) or a dedicated remote controller to send a movement instruction to the robot 60. The operation of the robot 60 is controlled by a movement instruction received from a portable terminal or a dedicated remote controller. In this case, the participants in the conference move the robot 60 by confirming the information for identifying the subject (information output on the display device 40) as described in the third embodiment described later. It is possible to determine the position to be made.

As described above, according to the present embodiment, the effects described in the first embodiment can be obtained. Further, in the present embodiment, the autonomously movable robot 60 automatically acquires the sub-image S in which an unspecified person is captured. Therefore, the participants in the meeting do not have to take any special action during the meeting to identify the unspecified person. That is, it is possible to save the trouble of identifying an unspecified person, and the convenience of the video conference system 1 is improved.

[Third Embodiment]
In the present embodiment, the processing when the imaging device 35 is incorporated in a portable device (for example, a smartphone, a tablet, a notebook computer, etc.) owned by the participants of the conference will be described. The present embodiment is the same as the first embodiment except for the following points.

FIG. 6 is a diagram showing a configuration example of the video conference system 1 of the third embodiment. As shown in FIG. 6, the image pickup apparatus 35 of the present embodiment is incorporated in a portable terminal 70 owned by a participant of the conference. The portable terminal 70 is, for example, a smartphone, a tablet, a notebook computer, or the like. Further, in the present embodiment, the processing execution unit 140 generates information for identifying the subject (unspecified person) of the image pickup apparatus 35 based on the position of the unspecified person detected by the position detecting unit 130. Specifically, the processing execution unit 140 sends information indicating that the person is an unspecified person to the main image M according to the position of the unspecified person in the main image M detected by the position detecting unit 130. Generate superimposed data. Then, the processing execution unit 140 outputs the information for identifying the subject of the image pickup device 35 generated in this way to the display device 40 (display device 40 provided at the shooting point of the main image M) (example). : Fig. 7).

FIG. 7 is a diagram showing an example of information for identifying a subject displayed on the display device 40. In FIG. 7, in addition to the information indicating the names of the identified participants, the character information "Unknown" is superimposed and displayed on the main image M. In the example of FIG. 7, the character information "Unknown" is information indicating that the person is an unspecified person. Which person should be photographed by using the image pickup apparatus 35 by confirming the information (character information "Unknown") as shown in FIG. 7 for the participants of the conference at the shooting point of the main image M. Can be grasped. Then, the participant himself / herself or another participant who is determined to be an unspecified person executes a shooting operation toward the participant who is determined to be an unspecified person with the imaging device 35 provided in the portable terminal 70. As a result, a sub-image S in which an unspecified person who has not been specified using the main image M is captured is generated. Here, a plurality of unspecified persons may exist in the main image M. In this case, information indicating which unspecified person the sub-image S taken by the portable terminal 70 corresponds to is required. Therefore, when a plurality of unspecified persons are present in the main image M, as an example, the portable terminal 70 may further accept an operation of designating the unspecified persons corresponding to the sub image S. For example, the portable terminal 70 may display the main image M on the display screen before or after shooting the sub image S, and may accept an operation of selecting an unspecified person from the main image M. Then, the portable terminal 70 transmits the generated sub-image S to the server device 10 according to the shooting operation or the further operation of the person who is operating the terminal. As a result, as described in the first embodiment, the person identification process using the sub-image S can be executed.

As described above, according to the present embodiment, the effects described in the first embodiment can be obtained.

[Fourth Embodiment]
The present embodiment has the same configuration as each of the above-described embodiments, except that it further has a function of automatically creating minutes.

FIG. 8 is a diagram showing a configuration example of the video conference system 1 according to the fourth embodiment. The video conference system 1 illustrated in FIG. 8 further includes a list creation unit 150, a voice acquisition unit 160, a speaker identification unit 170, and a minutes creation unit 180.

The list creation unit 150 creates a list of persons identified by the person identification process of the person identification unit 120. The list creation unit 150 operates as follows, for example. First, when a person is identified by the person identification process of the person identification unit 120, the list creation unit 150 acquires the result from the person identification unit 120. Then, the list creation unit 150 adds the person identification result acquired from the person identification unit 120 to the list held in the memory 1030 or the like. As a result, a list of participants in the conference held by using the video conference system 1 can be automatically generated.

The voice acquisition unit 160 acquires voice data of a conversation during a meeting generated by a microphone (not shown). The speaker identification unit 170 identifies a speaker regarding the voice data acquired by the voice acquisition unit 160. As an example, the speaker identification unit 170 makes a statement regarding the voice data acquired by the voice acquisition unit 160 by collating with the voiceprint data of each participant registered in advance in the storage device 1040 or the like before the conference is held. Can be identified. As another example, the speaker identification unit 170 analyzes an image (an image generated by the imaging device 30) acquired in synchronization with the voice data, so that the speaker identification unit 170 is a speaker regarding the voice data acquired by the voice acquisition unit 160. Can be identified. Specifically, the speaker identification unit 170 identifies a region of a person whose mouth is moving as a result of analyzing an image acquired in synchronization with voice data. Then, the speaker can be identified from the result of the person identification process for the area of the person whose mouth is moving. The minutes creation unit 180 generates the minutes data by associating the speaker identification result by the speaker identification unit 170 with the text data generated based on the voice data acquired by the voice acquisition unit 160. .. In addition, the minutes preparation unit 180 can add the list of persons generated by the list preparation unit 150 to the minutes data as participants in the meeting.

[Hardware configuration example]
The video conference system 1 of the present embodiment has the same hardware configuration as that of the first embodiment (example: FIG. 2). The storage device 1040 of the present embodiment further stores a program module for realizing the functions of the list creation unit 150, the voice acquisition unit 160, the speaker identification unit 170, and the minutes creation unit 180 described above. When the processor 1020 reads and executes these program modules on the memory 1030, each function of the above-described embodiment is realized.

[Processing flow]
The flow of processing executed by the video conference system 1 of the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart illustrating a flow of processing executed by the video conference system 1 of the fourth embodiment.

First, the voice acquisition unit 160 acquires the voice data of the conference (S202). The audio data of the conference is generated by the sound collecting device 50 provided at each point. The sound collecting device 50 is connected to the communication terminal 20. The voice acquisition unit 160 can communicate with the communication terminal 20 at each point via the network interface 1060 and acquire the voice data generated by the sound collector 50 at that point.

Then, the speaker identification unit 170 identifies the speaker regarding the voice data acquired by the voice acquisition unit 160 (S204). As an example, the speaker identification unit 170 can identify the speaker regarding the voice data acquired by the voice acquisition unit 160 as follows. First, the speaker identification unit 170 collates the voiceprint data of each participant registered in advance in the storage device 1040 or the like with the voice data, and obtains the voiceprint data whose degree of coincidence with the voiceprint of the voice data satisfies the standard. Identify. Then, the speaker identification unit 170 obtains the identification information (name of the person, ID assigned to each person, etc.) of the participating person associated with the specified voiceprint data, and thus the voice acquisition unit 160 uses the voice acquisition unit 160. The speaker of the acquired voice data can be identified. As another example, the speaker identification unit 170 can identify the speaker regarding the voice data acquired by the voice acquisition unit 160 as follows. First, the speaker identification unit 170 analyzes the image acquired by the image acquisition unit 110 in synchronization with the voice data. Specifically, the speaker identification unit 170 detects a region of a person's mouth from an image and determines whether or not the region (that is, the mouth) is moving between a plurality of images arranged in chronological order. .. Then, the speaker identification unit 170 acquires the result of the person identification process of the person identification unit 120 for the area of the person determined to be moving in the mouth area, and thus the voice acquired by the voice acquisition unit 160. The speaker of the data can be identified. Further, the speaker identification unit 170 identifies the speaker based on the direction in which the sound source is located and the detection position of the person (face) by combining a known speaker tracking method (for example, a sensor microphone and a face detection technique). Method) may be used to identify the speaker. Here, it is assumed that all the persons are specified by, for example, the processing described in each of the above-described embodiments.

The minutes creation unit 180 generates minutes data based on the processing results of the voice acquisition unit 160 and the speaker identification unit 170 (S206). Specifically, the minutes creation unit 180 converts the voice data acquired by the voice acquisition unit 160 into text data by using an API (Application Programming Interface) or the like that converts the voice data into text. In addition, the minutes preparation unit 180 acquires the speaker information (for example, the name of the speaker) of the voice data specified by the speaker identification unit 170. Then, the minutes creation unit 180 associates the text data generated from the voice data acquired by the voice acquisition unit 160 with the information of the person specified as the speaker regarding the voice data, and converts the minutes data into the minutes data. to add. Further, the minutes preparation unit 180 may read the list of participants in the meeting generated by the list preparation unit 150 and add the information of the participants to the minutes data.

As described above, according to the configuration of the present embodiment, the minutes of the meeting to be held using the video conference system 1 can be automatically created. As a result, it is possible to reduce the time and effort required for the participants in the meeting to prepare the minutes.

[Fifth Embodiment]
This embodiment differs from each of the above-described embodiments in the following points.

[System configuration example]
FIG. 10 is a diagram showing a configuration example of the video conference system 1 according to the fifth embodiment. As illustrated in FIG. 10, the video conference system 1 of the present embodiment does not include the position detection unit 130 and the image pickup device 35. Instead, the video conference system 1 of the present embodiment includes a voice acquisition unit 160 and a speaker identification unit 170. The operations of the voice acquisition unit 160 and the speaker identification unit 170 are the same as the operations described in the fourth embodiment. The processing execution unit 140 of the present embodiment identifies whether or not the unspecified person is a speaker related to the voice data by analyzing the main image M acquired in synchronization with the voice data. In the present embodiment, the personal authentication process based on the main image M and the personal authentication process based on the voice data are executed in parallel.

[Hardware configuration example]
FIG. 11 is a diagram illustrating a hardware configuration of the video conference system 1 according to the fifth embodiment. The hardware configuration illustrated in FIG. 11 differs from the hardware configuration illustrated in FIG. 2 in that the image pickup apparatus 35 is not provided. Further, the storage device 1040 of the present embodiment does not store the program module that realizes the function of the position detection unit 130. Instead, the storage device 1040 of the present embodiment further stores a program module for realizing the voice acquisition unit 160 and the speaker identification unit 170. Further, the program module of the processing execution unit 140 stored in the storage device 1040 of the present embodiment realizes the function of the processing execution unit 140 of the present embodiment described above.

[Processing flow]
A flow of processing executed by the video conference system 1 of the present embodiment will be described with reference to FIG. FIG. 12 is a flowchart illustrating a flow of processing executed by the video conference system 1 of the fifth embodiment. Here, the processes from S102 to S114 in FIG. 3 (personal authentication process based on the main image M) are performed in parallel.

First, the process execution unit 140 determines whether or not an unspecified person has been detected in the personal authentication process based on the main image M executed in parallel (S302). If an unspecified person is not detected (S302: NO), the subsequent processing is not executed.

On the other hand, when an unspecified person is detected (S302: YES), the speaker identification unit 170 starts a process of identifying the speaker related to the voice data by using the voice data acquired by the voice acquisition unit 160. (S304). The specific flow of the process of S304 is the same as the process of S304 of FIG.

Further, the processing execution unit 140 analyzes the main image M acquired in synchronization with the above-mentioned voice data, and determines whether or not the speaker specified in the processing of S304 matches the unspecified person ( S306). The processing execution unit 140 determines whether or not the speaker is an unspecified person based on whether or not the specific failure information is associated with the area of the person whose mouth is moving in the main image M. Can be done. When the speaker is an unspecified person (S306: YES), the processing execution unit 140 associates the person information of the participant specified as the speaker with the area of the unspecified person in the main image M (S308). By this association, the person information of the participating person is displayed on the main image M instead of the information indicating the unspecified person.

In the present embodiment, instead of using the image pickup apparatus 35 that generates the sub-image S, an unspecified person can be identified by using a voice authentication technique.

Although the embodiments of the present invention have been described above with reference to the drawings, these are examples of the present invention, and various configurations other than the above can be adopted.

Further, in the plurality of flowcharts used in the above description, a plurality of steps (processes) are described in order, but the execution order of the steps executed in each embodiment is not limited to the order of description. In each embodiment, the order of the illustrated steps can be changed within a range that does not hinder the contents. In addition, the above-described embodiments can be combined as long as the contents do not conflict with each other.

Some or all of the above embodiments may also be described, but not limited to:
1. 1.
An image acquisition means for acquiring an image of a person participating in the conference generated by the first imaging device, and
A person identification means that analyzes the image and executes a person identification process for identifying a person included in the image, and
A position detecting means for detecting the position of an unspecified person who could not be identified by the person identification process, and
A process execution means for executing a predetermined process for identifying the unspecified person using the detected position of the unspecified person, and a process executing means.
Video conference system with.
2.
The process executing means sets as the predetermined process a process of generating and outputting information for identifying a shooting position or a subject of a second image pickup device, which is a mobile image pickup device, based on the position of the unspecified person. Run and
The image acquisition means acquires an additional image generated by the second image pickup apparatus, and obtains an additional image.
The person identifying means analyzes the additional image generated by the second imaging device to identify the unspecified person.
1. 1. The video conference system described in.
3. 3.
The second imaging device is incorporated in an autonomously movable robot.
The processing execution means is
Based on the position of the unspecified person, information for specifying the shooting position of the second imaging device is generated.
By outputting the information for specifying the shooting position to the robot, the robot is guided to the position specified by the information to execute shooting.
2. The video conference system described in.
4.
The second imaging device is incorporated in a portable terminal owned by a participant of the conference.
The processing execution means is
Based on the position of the unspecified person, information for identifying the subject of the second imaging device is generated.
The information for identifying the subject is output to the display device.
2. The video conference system described in.
5.
Further provided with a list creation means for creating a list of persons identified by the person identification means.
1. 1. From 4. The video conference system according to any one of the above.
6.
Voice acquisition means to acquire voice data and
A speaker identifying means for identifying a speaker related to the voice data by analyzing the voice data or an image acquired in synchronization with the voice data.
Further provided is a minutes creation means for generating minutes data by associating the specific result of the speaker with the text data generated based on the voice data.
1. 1. From 5. The video conference system according to any one of the above.
7.
The computer
Acquire the image of the participants of the conference generated by the first imaging device,
The image is analyzed, a person identification process for identifying the person included in the image is executed, and the person identification process is executed.
Detects the position of an unspecified person that could not be specified by the person identification process,
Using the detected position of the unspecified person, a predetermined process for identifying the unspecified person is executed.
Video conference methods including that.
8.
The computer
A process of generating and outputting information for identifying a shooting position or a subject of a second image pickup device, which is a mobile image pickup device, based on the position of the unspecified person is executed as the predetermined process.
The additional image generated by the second image pickup apparatus is acquired, and the additional image is acquired.
The unspecified person is identified by analyzing the additional image generated by the second imaging device.
Including 7. The video conference method described in.
9.
The second imaging device is incorporated in an autonomously movable robot.
The computer
Based on the position of the unspecified person, information for specifying the shooting position of the second imaging device is generated.
By outputting the information for specifying the shooting position to the robot, the robot is guided to the position specified by the information to execute shooting.
Including 8. The video conference method described in.
10.
The second imaging device is incorporated in a portable terminal owned by a participant of the conference.
The computer
Based on the position of the unspecified person, information for identifying the subject of the second imaging device is generated.
The information for identifying the subject is output to the display device.
Including 8. The video conference method described in.
11.
The computer
Create a list of people identified by the person identification means,
Including 7. To 10. The video conference method according to any one of the above.
12.
The computer
Get audio data,
By analyzing the voice data or an image acquired in synchronization with the voice data, a speaker regarding the voice data can be identified.
Minutes data is generated by associating the specific result of the speaker with the text data generated based on the voice data.
Including 7. From 11. The video conference method according to any one of the above.
13.
To the computer, 7. From 12. A program for executing the video conference method described in any one of the above.

1 Video conferencing system 10 Server device 1010 Bus 1020 Processor 1030 Memory 1040 Storage device 1050 Input / output interface 1060 Network interface 110 Image acquisition unit 120 Person identification unit 130 Position detection unit 140 Processing execution unit 150 List creation unit 160 Voice acquisition unit 170 Speaker Specific unit 180 Minutes creation unit 20 Communication terminal 30 Imaging device 30 First imaging device 35 Imaging device 40 Display device 50 Sound collecting device 60 Robot 70 Portable terminal

Claims (10)

An information acquisition means for acquiring the first information for identifying the participants in the conference, which is generated by the first acquisition device.
A person identification means that analyzes the first information and executes a person identification process for identifying a person included in the first information , and a person identification means.
Specific information detection means for detecting specific information of an unspecified person that could not be specified by the person identification process, and
A process execution means for executing a predetermined process for acquiring a second information capable of identifying the unspecified person by analyzing the detected specific information .
Video conference system with. The person identification means analyzes the second information and executes a person identification process for identifying a person included in the second information.
The video conference system according to claim 1. The first information and the second information include biological information including at least one of facial information and voice.
The specific information includes at least one of position and voice.
The video conference system according to claim 1 or 2. The process executing means executes as the predetermined process a process of generating and outputting information for specifying a shooting position or a subject of a mobile imaging device based on the specific information of the unspecified person.
The information acquisition means acquires an additional image generated by the mobile imaging device, and obtains an additional image.
The person identifying means analyzes an additional image generated by the mobile imaging device to identify the unspecified person.
The video conference system according to any one of claims 1 to 3 . The mobile imaging device is incorporated in a robot that can move autonomously.
The processing execution means is
Based on the specific information of the unspecified person, information for specifying the shooting position of the mobile imaging device is generated.
By outputting the information for specifying the shooting position to the robot, the robot is guided to the position specified by the information to execute shooting.
The video conference system according to claim 4 . The mobile imaging device is incorporated in a portable terminal owned by a participant of the conference.
The processing execution means is
Based on the specific information of the unspecified person, information for identifying the subject of the mobile imaging device is generated.
The information for identifying the subject is output to the display device.
The video conference system according to claim 4 . Further provided with a list creation means for creating a list of persons identified by the person identification means.
The video conference system according to any one of claims 1 to 6 . Voice acquisition means to acquire voice data and
A speaker identifying means for identifying a speaker related to the voice data by analyzing the voice data or an image acquired in synchronization with the voice data.
Further provided is a minutes creation means for generating minutes data by associating the specific result of the speaker with the text data generated based on the voice data.
The video conference system according to any one of claims 1 to 7 . The computer
Acquires the first information for identifying the participants in the conference, which is generated by the first acquisition device.
The person identification process for identifying the person included in the first information by analyzing the first information is executed.
Detects specific information of an unspecified person that could not be specified by the person identification process,
Using the detected specific information , a predetermined process for acquiring a second information capable of identifying the unspecified person by analysis is executed.
Video conference methods including that. A program for causing a computer to execute the video conference method according to claim 9 .

Images