JP2017092675A - Information processing apparatus, conference system, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus, conference system, information processing method, and program that are capable of, with regard to a user participating in a conference, performing close-up on video of the user as intended.SOLUTION: An information processing apparatus comprises: an imaging unit for imaging video; an input unit for inputting voice; a recognition unit for recognizing a user from the video; an identification unit for identifying an arrangement pattern showing which direction, with respect to the information processing apparatus, the user is arranged; an acquisition unit for acquiring first information including information showing the user and the user's role associated with the information showing the user; a setting unit for setting, on the basis of the first information's role corresponding to the recognized user and second information associating the role with priority, priority for the user's role; an extraction unit for extracting a video region corresponding to a direction of a user having a role having higher priority, from the video, when voice is input from a plurality of directions; and a transmission unit for transmitting video data corresponding to the extracted video region.SELECTED DRAWING: Figure 7

Description

  The present invention relates to an information processing apparatus, a conference system, an information processing method, and a program.

  Video conferencing systems that conduct remote conferences with remote locations via a communication network such as the Internet have become widespread. In this video conference system, an image (video) of a participant in the conference room is taken using a terminal device of the video conference system in a conference room where one of the parties such as a participant conducting a remote conference (video conference) exists. In addition, the voice of the utterance is input, and the video data and the voice data are transmitted to the other party's terminal device. And it displays on the display of the other party's conference room, and outputs a voice by a speaker, thereby realizing a conference between remote locations in a state close to the actual conference.

  In the video conference system, a microphone is used to acquire the voice of the conference participant and a camera is used to acquire the video. However, since the camera has an angle of view, it is not possible to take pictures of participants outside the angle of view of the camera. In order to solve this problem, a method using a panoramic camera capable of photographing 360 degrees in all directions is known. On the other hand, since the microphone is usually omnidirectional, ambient sounds other than the participant's utterance are also collected. In order to solve this problem, there is a known method for collecting sound from participants by efficiently collecting sound from surrounding sounds by using a microphone array to provide directivity to the sound collection range of the microphone. Yes.

  As such a video conferencing system, the arrangement pattern of participants at the time of the conference is stored in advance, and the arrangement pattern pre-stored in the terminal device at the start of the meeting is selected according to the arrangement pattern of the participants In the conference terminal, a technique for setting the sound collection direction of the microphone array based on the selected arrangement pattern is disclosed (Patent Document 1).

  However, in the technique described in Patent Document 1, for example, when a plurality of participants speak at the same time, if the video (image) of any one of the participants is closed up (cut out of the video), There is a problem that the system cannot determine whether it is good and the video does not switch as intended.

  The present invention has been made in view of the above, and an information processing apparatus, a conference system, an information processing method, and an information processing apparatus that can close up a user's video as intended for users participating in a conference. The purpose is to provide a program.

  In order to solve the above-described problems and achieve the object, the present invention is an information processing apparatus, which is an imaging unit that captures an image, an input unit that inputs audio, and the image captured by the imaging unit A recognizing unit for recognizing the user from the image, and an arrangement pattern indicating in which direction the user recognized by the recognizing unit is arranged with respect to the information processing apparatus in the video imaged by the imaging unit. An acquisition unit that acquires first information including an identification unit to be identified, information indicating at least a user, and a role of the user associated with the information indicating the user, and a use recognized by the recognition unit The priority corresponding to the role of the user is set for the user based on the role of the first information corresponding to the user and the second information in which the role and the priority are associated in advance. In the setting part and the input part In the case where voice is input from a plurality of directions, priority is given to the direction of the user having a role with high priority among the directions of the users specified by the arrangement pattern in the plurality of directions. A cutout unit that cuts out a video area including the user corresponding to the direction from the video, and a transmission unit that transmits the video area cut out by the cutout unit.

  ADVANTAGE OF THE INVENTION According to this invention, a user's image | video can be closed up as intended about the user who has participated in the meeting.

FIG. 1 is a diagram illustrating an example of the overall configuration of a conference system according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating an example of the arrangement of a plurality of microphones and the arrangement of a panoramic camera of the information processing apparatus according to the embodiment. FIG. 4 is a diagram illustrating an example of a configuration of a conference information table according to the embodiment. FIG. 5 is a diagram illustrating an example of a configuration of a priority setting table according to the embodiment. FIG. 6 is a diagram illustrating an example of a configuration of a face recognition feature information table according to the embodiment. FIG. 7 is a diagram illustrating an example of a functional block configuration of the information processing apparatus according to the embodiment. FIG. 8 is a diagram illustrating an arrangement example of users participating in the conference. FIG. 9 is a flowchart illustrating an example of priority setting processing of the information processing apparatus according to the embodiment. FIG. 10 is a flowchart illustrating an example of the flow of the sound collection operation and video cutout operation of the information processing apparatus according to the embodiment.

  Hereinafter, embodiments of an information processing apparatus, a conference system, an information processing method, and a program according to the present invention will be described in detail with reference to FIGS. In addition, the present invention is not limited by the following embodiments, and constituent elements in the following embodiments are easily conceivable by those skilled in the art, substantially the same, and so-called equivalent ranges. Is included. Furthermore, various omissions, substitutions, changes, and combinations of the constituent elements can be made without departing from the scope of the following embodiments.

(Conference system configuration)
FIG. 1 is a diagram illustrating an example of the overall configuration of a conference system according to an embodiment. The configuration of the conference system 1 according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 1, the conference system 1 according to the present embodiment includes two or more information processing devices (information processing devices 10a, 10b,...), A conference server 20, and a reservation server 30. . The information processing apparatuses 10a and 10b can communicate with the conference server 20 and the reservation server 30 via the network 2 such as the Internet.

  The information processing devices 10a and 10b establish a session with other information processing devices based on the control of the conference server 20, and a conference terminal device that transmits and receives audio data and video data via the established session It is. Thereby, in the conference system 1, a video conference (hereinafter sometimes simply referred to as “conference”) between a plurality of information processing devices (10a, 10b,...) Is realized. Note that one or more information processing apparatuses (10a, 10b,...) Illustrated in FIG. 1 are simply referred to as “information processing apparatus 10” when referring to or collectively referring to any information processing apparatus.

  The conference server 20 is a server device that performs monitoring of whether or not each information processing device 10 is connected to the conference server 20, control of calling each information processing device 10 at the start of the conference, and control of information processing during the conference. .

  The reservation server 30 is a server device in which a user or the like who hosts the conference registers (reserves) the conference information (the date and time, the venue, the participating user, the role, the information processing device to be used, etc.) in advance. It is. The meeting information will be described later with reference to FIG. The reservation server 30 stores a priority setting table 1002 and a face recognition feature information table 1003 shown in FIGS. 5 and 6 to be described later. Details of each table will be described later. The reservation server 30 is connected to, for example, a management PC (Personal Computer) via the network 2 to register the conference information and add the priority setting table 1002 and the face recognition feature information table 1003. It is good also as what can perform setting, such as an update.

  The configuration of the conference system 1 shown in FIG. 1 is an example. For example, the conference server 20 and the reservation server 30 are separate server devices. However, the configuration is not limited to this, and the conference server 20 and the reservation server 30 are configured by one server device. It is good also as a thing.

(Hardware configuration of information processing device)
FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating an example of the arrangement of a plurality of microphones and the arrangement of a panoramic camera of the information processing apparatus according to the embodiment. Details of the hardware configuration of the information processing apparatus 10 according to the present embodiment will be described with reference to FIGS.

  As shown in FIG. 2, the information processing apparatus 10 according to the present embodiment includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and an auxiliary storage device 204. A media drive 205, an operation button 206, a power switch 207, a network I / F 208, an image sensor I / F 209, a panoramic camera 210, an audio I / F 211, a microphone array 212, a speaker 213, An output I / F 214 and an external device I / F 216 are provided.

  The CPU 201 is an integrated circuit that controls the operation of the entire information processing apparatus 10. The ROM 202 is a non-volatile storage device that stores programs such as firmware for the information processing apparatus 10. The RAM 203 is a volatile storage device used as a work area for the CPU 201.

  The auxiliary storage device 204 is a non-volatile storage device that stores various programs for realizing the operation of the information processing apparatus 10 and various data such as video data and audio data. The auxiliary storage device 204 is, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

  The media drive 205 is a device that controls reading and writing of data with respect to a recording medium 205a such as a flash memory.

  The operation button 206 is a button for performing a setting operation or the like for the information processing apparatus 10. The power switch 207 is a switch for switching on / off the power of the information processing apparatus 10.

  The network I / F 208 is an interface for communicating data using the network 2. The network I / F 208 is, for example, a NIC (Network Interface Card). The image sensor I / F 209 is an interface for transmitting video data to and from the panoramic camera 210 that captures a subject and obtains video data under the control of the CPU 201.

  The panoramic camera 210 is an imaging device that includes a lens and a solid-state imaging device that converts light into electric charges and converts an object image (video) into digital data. The panoramic camera 210 acquires video data of 360 degrees around. As described above, by acquiring 360-degree video data, it is possible to image all the users who participate in the conference around the information processing apparatus 10. The panoramic camera 210 is connected to the image sensor I / F 209. A CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) is used as the solid-state imaging device. Further, the panoramic camera 210 is installed, for example, at the center of the upper surface of the information processing apparatus 10 as shown in FIG.

  The audio I / F 211 is an interface that processes input / output of audio signals between the microphone array 212 that inputs audio and the speaker 213 that outputs audio, under the control of the CPU 201. The microphone array 212 is a sound collection device that inputs the voices of users participating in the conference. The microphone array 212 includes a plurality of microphones, and can implement directivity control that arbitrarily sets the sound collection direction according to the control of the CPU 201. The speaker 213 is a device that outputs sound in accordance with the control of the CPU 201. The microphone array 212 and the speaker 213 are connected to the audio I / F 211, respectively. Moreover, the microphone array 212 includes, for example, six microphones (212a to 212f) as illustrated in FIG. For example, as illustrated in FIG. 3, the microphones 212 a to 212 f are distributed on the upper surface of the housing of the information processing apparatus 10. These microphones 212a to 212f can switch the input operation valid or invalid, or switch the gain for amplifying the sound input by each of them to control sound in an arbitrary direction or range around 360 degrees. Can be collected. Note that the microphone array 212 is not limited to being configured with six microphones as illustrated in FIG. 3, and may have a plurality of microphones. Further, as shown in FIG. 3, each microphone of the microphone array 212 is arranged in a distributed manner in the housing of the information processing apparatus 10, but is not limited to this, and the microphones 212 a to 212 f are included. The unit of the microphone array 212 that is included may be configured separately from the housing of the information processing apparatus 10.

  The output I / F 214 is an interface for transmitting video data to the external display device 215 under the control of the CPU 201. The external device connection I / F 216 is an interface through which external devices such as an external camera, an external microphone, and an external speaker can be electrically connected by a USB (Universal Serial Bus) cable or the like.

  The display device 215 is a display device that displays images of users at other sites participating in the conference. The display device 215 is, for example, a CRT (Cathode Ray Tube) display, an LCD (Liquid Crystal Display), an organic EL (Organic Electro-Luminescence) display, or the like. The display device 215 is connected to the output I / F 214 by a cable 215a. The cable 215a may be an analog RGB (VGA) signal cable or a component video cable. The HDMI (registered trademark) (High-Definition Multimedia Interface) or DVI (Digital Video Interactive) signal may be used as the cable 215a. It may be a cable for use.

  CPU 201, ROM 202, RAM 203, auxiliary storage device 204, media drive 205, operation button 206, power switch 207, network I / F 208, image sensor I / F 209, audio I / F 211, output I / F 214, and external device I described above The / F216 is connected to be communicable with each other via a bus 217 such as an address bus and a data bus.

  Note that the hardware configuration of the information processing apparatus 10 is not limited to the configuration shown in FIG. For example, the media drive 205 may not be provided.

(Various information tables)
FIG. 4 is a diagram illustrating an example of a configuration of a conference information table according to the embodiment. The conference information table 1001 stored in the reservation server 30 will be described with reference to FIG.

  The reservation server 30 shown in FIG. 1 stores a conference information table 1001 for managing registered conference information as described above. As shown in FIG. 4, in the conference information table 1001, for example, for each conference identification information for identifying conference information, the date and time of the conference, the location, the user identification information for identifying the user, the role, and the conference are used. Used terminals (information processing apparatus 10) are managed in association with each other. Here, the role is a role in the conference assigned to a user who participates in the conference. For example, the role includes a chairperson, minutes (users who take minutes), board (users who make board), and normal participants other than the chair, minutes, and board. The used terminal is, for example, identification information that uniquely identifies a conference terminal (information processing apparatus 10) used in the conference room.

  For example, in the conference information table 1001 shown in FIG. 4, as the conference information whose conference identification information is “002”, the date and time of the meeting is “2015/11/14 09:30” and the meeting place is “Meeting room 2-F”. The user identification information is “10507, 20311”, the role is “chairperson, participant”, and the terminal used is “VD5011”.

  Note that the conference information table 1001 shown in FIG. 4 is information in a table format, but is not limited to this, and any format can be used as long as the values of the fields of the table can be managed in association with each other. It may be information.

  FIG. 5 is a diagram illustrating an example of a configuration of a priority setting table according to the embodiment. The priority setting table 1002 stored in the reservation server 30 will be described with reference to FIG.

  The reservation server 30 shown in FIG. 1 stores a priority setting table 1002 (an example of second information) for managing priorities for roles. As shown in FIG. 5, in the priority setting table 1002, the priority is associated with each role of the conference and managed. Here, the priority is a value that determines a priority order for cutting out the video of the user assigned to each role in the video data captured by the panoramic camera 210 during the conference.

  For example, in the priority setting table 1002 shown in FIG. 5, the roles “chairman”, “minutes”, “participants”, “whiteboard”, and “others” have priority “3”, “ “2”, “1”, “4” and “0” are assigned. In the example of FIG. 5, the higher the priority value, the higher the priority. That is, the priority setting table 1002 shown in FIG. 5 indicates that the priority of the role “whiteboard” is the highest at “4” and the priority of the role “other” is the lowest at “0”.

  The priority setting table 1002 shown in FIG. 5 is information in a table format, but is not limited to this, and any format can be used as long as roles and priorities can be managed in association with each other. It may be information.

  FIG. 6 is a diagram illustrating an example of a configuration of a face recognition feature information table according to the embodiment. The face recognition feature information table 1003 stored in the reservation server 30 will be described with reference to FIG.

  The reservation server 30 shown in FIG. 1 stores a face recognition feature information table 1003 for managing feature information of face images corresponding to users. As shown in FIG. 6, in the face recognition feature information table 1003, feature information of the face image of the user is managed in association with each piece of user identification information for identifying the user. Here, the feature information of the user's face image is information including, for example, the contour of the user's face, the shape and relative position of each part such as the eyes, nose, chin, and cheekbone. This is information for recognizing the user's face from the image picked up by.

  For example, in the face recognition feature information table 1003 shown in FIG. 6, feature information “{72, 123, −3,..., −110, 56, 219}” is associated with the user identification information “20391”. It has been.

(Function block configuration of information processing device)
FIG. 7 is a diagram illustrating an example of a functional block configuration of the information processing apparatus according to the embodiment. FIG. 8 is a diagram illustrating an arrangement example of users participating in the conference. Details of the functional block configuration of the information processing apparatus 10 according to the present embodiment will be described with reference to FIGS.

  As shown in FIG. 7, the information processing apparatus 10 according to the present embodiment includes an acquisition unit 101, a recognition unit 102, a specifying unit 103, a cutout unit 104, a switching unit 105, a setting unit 106, Transmission unit 107, reception unit 108, imaging control unit 109, display control unit 110, audio output control unit 111, input unit 112, storage unit 113, operation unit 114, communication unit 115, imaging Unit 116, display unit 117, and audio output unit 118.

  The acquisition unit 101 is a functional unit that acquires conference information (first information) from the reservation server 30 via the communication unit 115 and the network 2. Specifically, the acquisition unit 101 sends, for example, an acquisition request for acquiring conference information, and information on the date and time of the conference, the location of the conference, and the terminal in use via the communication unit 115 and the network 2. Send to. When the reservation server 30 receives the acquisition request, the reservation server 30 refers to the conference information table 1001 shown in FIG. 4 to obtain the user identification information and the role corresponding to the received date and time, the venue, and the terminal used. The data is transmitted to the acquisition unit 101 via the unit 115. For example, the acquisition unit 101 is realized by the CPU 201 illustrated in FIG. 2 executing a program.

  The recognition unit 102 is a functional unit that recognizes a user (hereinafter also referred to as “face recognition”) from one or more user face images included in the image captured by the imaging unit 116. Specifically, for example, the recognition unit 102 first detects each of the facial contour, eyes, nose, chin, cheekbones, and the like from one or more user face images included in the image captured by the imaging unit 116. Feature information including the shape and relative position of the part is extracted. Next, the recognizing unit 102 transmits the extracted feature information for one or more users to the reservation server 30 via the communication unit 115 and the network 2. When the reservation server 30 receives the feature information extracted from the recognition unit 102, the reservation server 30 refers to the face recognition feature information table 1003 shown in FIG. 6 and user identification information corresponding to the feature information that matches the received feature information. Is transmitted to the recognition unit 102 via the network 2 and the communication unit 115. The recognizing unit 102 recognizes users participating in the conference by receiving the user identification information. That is, the user indicated by the user identification information received by the recognition unit 102 is the user who actually participates in the conference. Here, the feature information extracted by the recognition unit 102 and the feature information included in the face recognition feature information table 1003 match not only when the two feature information completely match, but also with the two feature information. A case where it can be determined that the feature information indicates substantially the same user may be included. The recognition unit 102 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The identification unit 103 identifies the position of the user recognized by the recognition unit 102 in the image captured by the imaging unit 116, and identifies the arrangement pattern of the user in the conference room of the user participating in the conference. It is. For example, in the situation illustrated in FIG. 8A, an image in 360 degrees omnidirectional (hereinafter referred to as “panoramic image”) captured by the imaging unit 116 of the information processing apparatus 10 placed on the desk 40 may be used. ), For the users 60a to 60e recognized by the recognition unit 102, the user 60a is in the area P1, the user 60b is in the area P2, the user 60c is in the area P3, and the user 60d is An arrangement pattern indicating that the user 60e is located in the area P6 and the whiteboard 50 is located in the area P7 is specified in the area P5. In the situation shown in FIG. 8B, in the panoramic image captured by the imaging unit 116, the specifying unit 103 sets the user 61 a to the region P <b> 2 for the users 61 a to 61 d recognized by the recognition unit 102. An arrangement pattern indicating that the user 61b is located in the area P3, the user 61c is located in the area P4, the user 61d is located in the area P5, and the whiteboard 50 is located in the area P7 is specified. That is, specifying the arrangement pattern specifies which user or whiteboard is arranged in which direction with respect to the information processing apparatus 10. The specifying unit 103 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program. Note that a user, a whiteboard, or the like in which the direction arranged with respect to the information processing apparatus 10 is specified by the arrangement pattern may be referred to as “user etc.”.

  In order to specify the position of the whiteboard 50, it is assumed that the positional relationship between the information processing apparatus 10 and the whiteboard 50 is stored in the storage unit 113 in a state where the information processing apparatus 10 is installed on the desk 40. Also good. Alternatively, the recognition unit 102 may recognize the whiteboard 50 in the panoramic image by a known method such as template matching based on the reference image of the whiteboard 50 instead of only the user's face.

  In addition, an image captured by the imaging unit 116 is sometimes referred to as a “panoramic image”. However, when an image is shown as an object captured by the imaging unit 116, it may be referred to as a “panoramic image”. However, the video is a concept including an image.

  Based on the priority corresponding to the role assigned to the user, the cutout unit 104 selects a specific user or the like from the panoramic video (in the example of FIG. 8, for example, any one of the plurality of users and the whiteboard 50). This is a functional part that cuts out the video area. The cutting unit 104 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  Based on the priority corresponding to the role assigned to the user, the switching unit 105 changes the sound collection direction of the input unit 112 to a specific user or the like (in the example of FIG. 8, for example, a plurality of users and whiteboards). It is a functional unit that switches so that any one of 50) is located. The switching unit 105 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The setting unit 106 detects the user recognized by the recognition unit 102 based on the operation information from the operation unit 114 that has received the user's operation input or based on the conference information acquired by the acquisition unit 101. This is a function unit for setting the priority. The setting unit 106 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The transmission unit 107 transmits the video data corresponding to the video area cut out by the cut-out unit 104 and the audio data input by the input unit 112 via the communication unit 115 and the network 2 to information processing apparatuses at other bases. 10 is a functional unit that transmits data to 10. Specifically, for example, the transmission unit 107 encodes video data and audio data and transmits the encoded data to the information processing apparatus 10 at another site. Here, a known method may be used as the encoding method. For example, H.M. H.264 / AVC, or H.264. A compression coding technique such as H.264 / SVC may be used. The transmission unit 107 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The receiving unit 108 is a functional unit that receives video data and audio data received from the information processing apparatus 10 at another site via the network 2 and the communication unit 115. Specifically, for example, the receiving unit 108 decodes received video data and audio data, sends the decoded video data to the display control unit 110, and sends the decoded audio data to the audio output control unit 111. Here, a known method may be used as a decoding method. The receiving unit 108 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The imaging control unit 109 is a functional unit that controls the operation of the imaging unit 116. Specifically, for example, the imaging control unit 109 controls the start and stop operations of imaging performed by the imaging unit 116 and acquires a panoramic image captured by the imaging unit 116. The imaging control unit 109 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The display control unit 110 is a functional unit that performs control to display various images on the display unit 117. The display control unit 110 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The audio output control unit 111 is a functional unit that controls the audio output unit 118 to output various types of audio. The audio output control unit 111 is realized, for example, when the CPU 201 illustrated in FIG. 2 executes a program.

  The input unit 112 is a functional unit that inputs voice. The input unit 112 inputs sound in a specific sound collection direction according to the control of the switching unit 105. The input unit 112 is realized by, for example, the microphone array 212 illustrated in FIG.

  The storage unit 113 is a functional unit that stores various programs for realizing the operation of the information processing apparatus 10, video data, audio data, and information such as the arrangement pattern specified by the specifying unit 103. The storage unit 113 is realized by, for example, the RAM 203 and the auxiliary storage device 204 illustrated in FIG.

  The operation unit 114 is a functional unit that receives various operation inputs of a user (for example, a participant in a conference). The operation unit 114 is realized by, for example, the operation button 206 and the power switch 207 shown in FIG. Note that the operation unit 114 is not limited to the operation button 206 and the power switch 207 illustrated in FIG. 2, and may be realized by a mouse, a keyboard, a touch panel, or the like.

  The communication unit 115 is a functional unit that performs data communication with the other information processing apparatus 10, the conference server 20, and the reservation server 30 via the network 2. The communication unit 115 is realized by, for example, the network I / F 208 shown in FIG.

  The imaging unit 116 is a functional unit that captures panoramic images or panoramic images in all directions of 360 degrees. The imaging unit 116 is realized by, for example, the panoramic camera 210 illustrated in FIG.

  The display unit 117 is a functional unit that displays various images under the control of the display control unit 110. The display unit 117 is realized by, for example, the display device 215 illustrated in FIG.

  The sound output unit 118 is a functional unit that outputs various sounds according to the control of the sound output control unit 111. The audio output unit 118 is realized by, for example, the speaker 213 illustrated in FIG.

  Note that the acquisition unit 101, the recognition unit 102, the identification unit 103, the clipping unit 104, the switching unit 105, the setting unit 106, the transmission unit 107, the reception unit 108, the imaging control unit 109, and the display of the information processing apparatus 10 illustrated in FIG. The control unit 110, the audio output control unit 111, the input unit 112, the storage unit 113, the operation unit 114, the communication unit 115, the imaging unit 116, the display unit 117, and the audio output unit 118 are conceptually shown functions. Thus, the present invention is not limited to such a configuration. For example, a plurality of functional units illustrated as independent functional units in the information processing apparatus 10 illustrated in FIG. 7 may be configured as one functional unit. On the other hand, in the information processing apparatus 10 illustrated in FIG. 7, the functions of one function unit may be divided into a plurality of functions and configured as a plurality of function units.

  In addition, the acquisition unit 101, the recognition unit 102, the identification unit 103, the cutout unit 104, the switching unit 105, the setting unit 106, the transmission unit 107, the reception unit 108, the imaging control unit 109, the display control unit 110, and the information processing apparatus 10 A part or all of the audio output control unit 111 may be realized by a hardware circuit such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) instead of a program that is software.

(Priority setting process)
FIG. 9 is a flowchart illustrating an example of priority setting processing of the information processing apparatus according to the embodiment. With reference to FIG. 9, the flow of the priority setting process of the information processing apparatus 10 according to the present embodiment will be described.

<Step S11>
First, a user who wants to participate in the conference operates the operation unit 114 of the information processing apparatus 10 to turn on the power of the information processing apparatus 10 and captures a surrounding image (panoramic image) with the imaging unit 116. Operation input is performed. Upon receiving operation information for capturing a panoramic image from the operation unit 114, the imaging control unit 109 causes the imaging unit 116 to capture a panoramic image. Then, the process proceeds to step S12.

<Step S12>
The recognizing unit 102 determines the shape and relative position of each part such as the face outline, eyes, nose, chin, and cheekbone from one or more user face images included in the panoramic image captured by the image capturing unit 116. Extract feature information. Next, the recognizing unit 102 transmits the extracted feature information for one or more users to the reservation server 30 via the communication unit 115 and the network 2. When the reservation server 30 receives the feature information extracted from the recognition unit 102, the reservation server 30 refers to the face recognition feature information table 1003 shown in FIG. 6 and user identification information corresponding to the feature information that matches the received feature information. Is transmitted to the recognition unit 102 via the network 2 and the communication unit 115. The recognition unit 102 recognizes (face recognition) a user who participates in the conference by acquiring user identification information.

  The specifying unit 103 specifies the position of the user recognized by the recognition unit 102 in the panoramic image captured by the imaging unit 116, and specifies the arrangement pattern of the user in the conference room of the user participating in the conference. To do. The identifying unit 103 causes the storage unit 113 to store information on the identified arrangement pattern. Then, the process proceeds to step S13.

<Step S13>
The acquisition unit 101 acquires conference information from the reservation server 30 via the communication unit 115 and the network 2. Specifically, the acquisition unit 101 transmits an acquisition request for acquiring conference information and information on the date and time of the conference, the location of the conference, and the terminal used to the reservation server 30 via the communication unit 115 and the network 2. To do. When the reservation server 30 receives the acquisition request, the reservation server 30 refers to the conference information table 1001 shown in FIG. 4 and stores the user identification information and the role corresponding to the received date and time, the venue, and the terminal used in the network 2 and the communication unit 115. To the acquisition unit 101. And the acquisition part 101 acquires the user identification information and role of the user who participates in a meeting as meeting information of a meeting to be held from now. When meeting information of a meeting to be held can be obtained by the obtaining unit 101 (step S13: Yes), the process proceeds to step S14, and when it cannot be obtained (step S13: No), the process proceeds to step S15.

<Step S14>
The setting unit 106 sends an acquisition request for acquiring the priority corresponding to the role in the conference information acquired by the acquiring unit 101 and the information on the role via the communication unit 115 and the network 2 to the reservation server 30. Send to. When receiving the acquisition request, the reservation server 30 refers to the priority setting table 1002 shown in FIG. 5 and transmits the priority corresponding to the received role to the setting unit 106 via the network 2 and the communication unit 115. . The setting unit 106 also includes user identification information in the conference information acquired by the acquisition unit 101 and user identification information acquired by face recognition by the recognition unit 102 (use of users who actually participate in the conference). The received priority is set for the user based on the role corresponding to the user whose user ID matches the user identification information. Then, the process proceeds to step S18.

  In addition, when the user identification information acquired by the face recognition by the recognition unit 102 is not included in the user identification information in the conference information acquired by the acquisition unit 101, that is, the usage not registered in the conference information. When the person exists in the actual meeting room, the setting unit 106 may set a default role (for example, “participant”). Alternatively, the setting unit 106 may determine that a user who is not registered in the conference information does not participate in the actual conference, and sets the role to “other”. Alternatively, when a user who is not registered in the conference information exists in the actual conference room, the user manually performs a role setting operation on the unregistered user via the operation unit 114. It may be done.

<Step S15>
An operation input in which the user manually sets a role from the operation unit 114 to the user whose face is recognized by the recognition unit 102 when the acquisition unit 101 cannot acquire the meeting information of the meeting to be held (Step S15: Yes), the process proceeds to step S16. On the other hand, when the user does not perform the operation input for manually setting the role from the operation unit 114 to the user whose face is recognized by the recognition unit 102 (step S15: No), the process proceeds to step S17.

<Step S16>
The setting unit 106 sets a role for the user whose face is recognized by the recognition unit 102 based on the role setting operation input from the operation unit 114 by the user, and obtains a priority corresponding to the role. An acquisition request for performing the request and information on its role are transmitted to the reservation server 30 via the communication unit 115 and the network 2. When receiving the acquisition request, the reservation server 30 refers to the priority setting table 1002 shown in FIG. 5 and transmits the priority corresponding to the received role to the setting unit 106 via the network 2 and the communication unit 115. . The setting unit 106 sets the received priority for the user based on the role corresponding to the user whose face is recognized by the recognition unit 102. Then, the process proceeds to step S18.

<Step S17>
The setting unit 106 sets a role for the user whose face is recognized by the recognition unit 102 based on the default role setting, an acquisition request for acquiring the priority corresponding to the role, and the role Information is transmitted to the reservation server 30 via the communication unit 115 and the network 2. When receiving the acquisition request, the reservation server 30 refers to the priority setting table 1002 shown in FIG. 5 and transmits the priority corresponding to the received role to the setting unit 106 via the network 2 and the communication unit 115. . The setting unit 106 sets the received priority for the user based on the role corresponding to the user whose face is recognized by the recognition unit 102. Here, the default role setting means, for example, that the role is set to “chairperson” preferentially for the user existing in the area P1 shown in FIG. 8, and is preferential to the user existing in the area P2. The role is set to “minutes”, the role is set to “board” preferentially for the users existing in the area P6, and the roles of other users are set to “participants”. This is a predetermined role setting. Then, the process proceeds to step S18.

<Step S18>
The information processing apparatus 10 starts the video conference after the setting unit 106 sets the role and priority for the user whose face is recognized. Specifically, the information processing apparatus 10 proceeds to processing of a sound collection operation and a video cut-out operation shown in FIG.

  The priority setting process is executed by the information processing apparatus 10 through the operations in steps S11 to S18.

  In the priority setting process shown in FIG. 9, whether the role and priority are set based on the conference information or whether the role and priority is set based on the role setting operation by the user is determined by the reservation server 30. The conference information table 1001 is determined based on the presence / absence of corresponding conference information, but is not limited to this. For example, the information processing apparatus 10 can set either the manual mode or the automatic mode. In the manual mode, the user can set the role by the user's role setting operation without referring to the conference information table 1001 of the reservation server 30. In the case of the automatic mode, it may be set with reference to the conference information table 1001 of the reservation server 30.

  In addition, among the priority setting processes shown in FIG. 9, the processes in steps S11 to S17 may be re-executed every predetermined time or when a predetermined condition is satisfied even during the meeting. When the predetermined condition is satisfied, for example, when the position of the user whose face is recognized by the recognition unit 102 in the panoramic image captured by the image pickup unit 116 is moved, the use of the face recognized by the recognition unit 102 last time is used. For example, when the user disappears, or when a user other than the user whose face was previously recognized by the recognition unit 102 was recognized as a face.

(Sound collection operation and video cut-out operation)
FIG. 10 is a flowchart illustrating an example of the flow of the sound collection operation and video cutout operation of the information processing apparatus according to the embodiment. With reference to FIG. 10, the flow of the sound collection operation and video cutout operation of the information processing apparatus 10 according to the present embodiment will be described.

<Step S31>
First, the switching unit 105 switches the sound collection direction of the input unit 112 to be 360 degrees in all directions. Then, the process proceeds to step S32.

<Step S32>
The clipping unit 104 captures an image area of a user having a role with the highest priority among the users whose arrangement direction is determined by the arrangement pattern specified by the specifying unit 103 by the imaging unit 116. Cut out from the panoramic video. For example, when the priority setting table 1002 is set to the content shown in FIG. 5, the priority of the role “whiteboard” is the highest at “4”, so that the cutout unit 104 has the arrangement direction determined by the arrangement pattern. The video area including the whiteboard is cut out from the panorama video. Then, the process proceeds to step 33.

  In step S32, the cutout unit 104 cuts out a video area of a user or the like having a role with the highest priority. However, the present invention is not limited to this. For example, the setting as to which user or the like is to be cut out may be set independently of the priority.

<Step S33>
The transmission unit 107 encodes video data corresponding to the video area cut out by the cut-out unit 104 and transmits the encoded video data to the information processing apparatus 10 at another base via the communication unit 115 and the network 2. Then, control goes to a step S34.

<Step S34>
The input unit 112 starts accepting voice input. Then, the process proceeds to step S35.

<Step S35>
The information processing apparatus 10 determines whether or not the video conference has ended. For example, the information processing apparatus 10 determines whether the user has performed an operation for ending the conference from the operation unit 114. When the video conference is finished (step S35: Yes), the sound collection operation and the video cutout operation are finished. When the video conference is not finished (step S35: No), the process proceeds to step S36.

<Step S36>
When the voice is input by the input unit 112 (step S36: Yes), the process proceeds to step S37, and when the voice is not input by the input unit 112 (step S36: No), the process returns to step S31.

<Step S37>
The input unit 112 determines whether or not the input sound is sound from a plurality of directions. When audio from a plurality of directions is input by the input unit 112 (step S37: Yes), the process proceeds to step S38, and when audio from one direction is input by the input unit 112 (step S37: No), Control goes to step S40.

<Step S38>
When voices from a plurality of directions are input by the input unit 112, the switching unit 105 identifies and identifies users corresponding to a plurality of directions from the arrangement pattern of the users and the like identified by the identifying unit 103. The sound collection direction of the input unit 112 is switched to the direction of the user who has the highest priority role among the selected users.

  Here, for example, when the sound collection direction of the input unit 112 is switched to the direction of a specific user by the switching unit 105 and then a sound is input from a new direction by the input unit 112, the switching unit 105 The direction of the user having a role with a higher priority by comparing the priority of the role of the user corresponding to the sound collection direction of the current input unit 112 with the priority of the role of the user corresponding to the new direction. In addition, the sound collection direction of the input unit 112 may be switched. The same applies to step S40 described later.

  Note that although the switching unit 105 switches the sound collection direction of the input unit 112 to the direction of the user having the highest priority among the identified users, the present invention is not limited to this. That is, the switching unit 105 switches the sound collection direction of the input unit 112 with priority given to the direction of a user having a higher priority role than the priority of other user roles among the identified users. It may be a thing.

  In addition, when the priority of the roles that the users corresponding to a plurality of directions specified by the arrangement pattern have the same priority, for example, the switching unit 105 is determined randomly among the users having the roles of the same priority, for example. What is necessary is just to switch the sound collection direction of the input part 112 to a user's direction. Alternatively, the switching unit 105 switches the sound collection direction of the input unit 112 with priority given to the direction of the user whose voice has been input first by the input unit 112 among the users having equal priority roles. Also good.

  The input unit 112 inputs sound from the sound collection direction switched by the switching unit 105. Then, the process proceeds to step S39.

<Step S39>
The cutout unit 104 cuts out a video area including the user corresponding to the sound collection direction switched by the switching unit 105 from the panoramic video captured by the imaging unit 116. Then, the process proceeds to step S42.

<Step S40>
When the voice from one direction is input by the input unit 112, the switching unit 105 identifies and identifies the user corresponding to the one direction from the arrangement pattern of the users and the like identified by the identifying unit 103. The sound collection direction of the input unit 112 is switched to the user direction. The input unit 112 inputs sound from the sound collection direction switched by the switching unit 105. Then, the process proceeds to step S41.

  Note that the switching unit 105 determines that the role of the user corresponding to the one direction specified by the arrangement pattern and the priority thereof are “other” and “0” as in the priority setting table 1002 shown in FIG. ”, It is determined that the user does not participate in the conference, and the sound collection direction is not switched to the direction of the user, that is, the sound uttered by the user may not be collected. . In this case, the cutout unit 104 may not cut out the video area including the user from the panoramic video also in step S41 described later.

<Step S41>
The cutout unit 104 cuts out a video area including the user corresponding to the sound collection direction switched by the switching unit 105 from the panoramic video captured by the imaging unit 116. Then, the process proceeds to step S42.

<Step S42>
The transmission unit 107 encodes the video data corresponding to the video region cut out by the cut-out unit 104 and the audio data input by the input unit 112, and transmits information on other bases via the communication unit 115 and the network 2. It transmits to the processing apparatus 10. And it returns to step S35 and operation | movement is repeated.

  Through the operations in steps S31 to S42 described above, the information processing apparatus 10 performs a sound collection operation and a video cutout operation.

  As described above, in the information processing apparatus 10 according to the present embodiment, roles and priorities are set in advance for users who participate in a video conference, and priority is given when a plurality of users speak. The sound collection direction of the input unit 112 is switched to the direction of a user having a high role, and a video area including the user corresponding to the sound collection direction is cut out from the panoramic image. Then, the transmission unit 107 encodes the video data corresponding to the video region cut out by the cut-out unit 104 and the audio data input by the input unit 112 and transmits the encoded data to the information processing apparatus 10 at another base. Yes. As a result, even when a plurality of users are speaking at the same time during the conference, the voices of the users with high priority are collected, and an image including the users is cut out to the information processing apparatus 10 at another base. Since it is transmitted, it is possible to close up the video of the user as intended for the user participating in the conference.

  Further, since the conference information is registered in advance in the reservation server 30, it is not necessary to set the role of the user who participates in the conference again, so that labor can be saved and the conference can be started smoothly.

  In addition, before the start of the conference, the imaging unit 116 captures a panoramic image, the recognition unit 102 recognizes the user participating in the conference, and the specifying unit 103 specifies the user arrangement pattern. It is not necessary to perform operations such as face recognition and arrangement pattern identification every time, and the load on the CPU 201 can be reduced.

  In addition, although the imaging part 116 shall be implement | achieved by the panorama camera 210 and shall take a panoramic image or a panoramic image, it is not necessarily limited to this. That is, when the imaging range does not need to be 360 degrees in all directions, it is not necessary to use a panoramic camera. For example, even an imaging device (camera) having an angle of view that covers a necessary imaging range can be used. Good. In this case, it is only necessary to cut out the video and set the sound collection direction within the range of the angle of view that can be captured by the imaging apparatus.

  In the above-described embodiment, when at least one of the functional units of the information processing apparatus 10 is realized by executing a program, the program is provided by being incorporated in advance in a ROM or the like. The program executed by the information processing apparatus 10 according to the above-described embodiment is a file in an installable format or an executable format, and is a CD-ROM (Compact Disc Read Only Memory), a flexible disk (FD), a CD. -R (Compact Disk-Recordable) or DVD (Digital Versatile Disc) may be stored and provided in a computer-readable storage medium. Further, the program executed by the information processing apparatus 10 according to the above-described embodiment may be configured to be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. Further, the program executed by the information processing apparatus 10 according to the above-described embodiment may be configured to be provided or distributed via a network such as the Internet. In addition, the program executed by the information processing apparatus 10 according to the above-described embodiment has a module configuration including at least one of the above-described functional units. As actual hardware, the CPU 201 includes the above-described storage device. By reading and executing the program from the ROM 202 and the auxiliary storage device 204 (for example, the ROM 202 and the auxiliary storage device 204), the above-described functional units are loaded on the main storage device (for example, the RAM 203) and generated.

DESCRIPTION OF SYMBOLS 1 Conference system 2 Network 10, 10a, 10b Information processing apparatus 20 Conference server 30 Reservation server 40 Desk 50 White board 60a-60e User 61a-61d User 101 Acquisition part 102 Recognition part 103 Identification part 104 Cutting part 105 Switching part DESCRIPTION OF SYMBOLS 106 Setting part 107 Transmission part 108 Reception part 109 Imaging control part 110 Display control part 111 Audio | voice output control part 112 Input part 113 Storage part 114 Operation part 115 Communication part 116 Imaging part 117 Display part 118 Audio | voice output part 201 CPU
202 ROM
203 RAM
204 Auxiliary storage device 205 Media drive 205a Recording medium 206 Operation button 207 Power switch 208 Network I / F
209 Image sensor I / F
210 Panorama Camera 211 Audio I / F
212 Microphone array 212a to 212f Microphone 213 Speaker 214 Output I / F
215 Display device 215a Cable 216 External device I / F
217 Bus 1001 Conference information table 1002 Priority setting table 1003 Face recognition feature information table P1 to P7 area

JP 2007-274463 A

Claims (12)

An information processing apparatus,
An imaging unit for imaging video;
An input unit for inputting voice;
A recognition unit for recognizing a user from the video imaged by the imaging unit;
A specifying unit for specifying an arrangement pattern indicating a direction in which the user recognized by the recognition unit is arranged with respect to the information processing device in the video imaged by the imaging unit;
An acquisition unit that acquires first information including at least information indicating a user and a role of the user associated with the information indicating the user;
Based on the role of the first information corresponding to the user recognized by the recognition unit, and the second information in which the role and the priority are associated in advance, the user's role with respect to the user A setting unit for setting the priority corresponding to
When voice is input from a plurality of directions by the input unit, priority is given to the direction of a user having a higher priority among the user directions specified by the arrangement pattern in the plurality of directions. A cutout unit that cuts out a video area including the user corresponding to the direction from the video;
A transmission unit for transmitting the video region cut out by the cut-out unit;
An information processing apparatus comprising: The input unit is capable of switching the direction of sound collection,
When voice is input from a plurality of directions by the input unit, among the users in the plurality of directions specified from the arrangement pattern, the input unit is set in the direction of the user having the highest priority role. The information processing apparatus according to claim 1, further comprising a switching unit that switches a sound collection direction of the sound. The switching unit is a user corresponding to the different direction when sound is input from a direction different from the one direction in a state where the input unit is switched to the sound collection direction by the input unit. If the priority of the role of the user is higher than the priority of the role of the user corresponding to the one direction, the sound collection direction of the input unit is switched to the different direction,
The information processing apparatus according to claim 2, wherein the cutout unit cuts out a video region including a user corresponding to the different direction from the video.   The information processing apparatus according to any one of claims 1 to 3, wherein the cutout unit cuts out a video region corresponding to a specific direction from the video when no audio is input from the input unit.   The cutout unit cuts out a video region corresponding to a direction of a role having the highest priority in the second information as the specific direction from the video when no audio is input from the input unit. The information processing apparatus described in 1. When the voice is input from a plurality of directions by the input unit, and the roles of the users in the plurality of directions identified from the arrangement pattern are equal to each other, the switching unit Priority is given to the voice input to the input unit earlier, and the sound collection direction of the input unit is switched to the direction of the user corresponding to the voice.
The information processing apparatus according to claim 2, wherein the cutout unit cuts out a video area including a user corresponding to the sound collection direction from the video. The switching unit switches the sound collection direction of the input unit to the direction of the user when the priority of the role of the user corresponding to the voice input by the input unit is the lowest in the second information. Without
The information processing apparatus according to claim 2, wherein the cutout unit does not cut out a video area including the user from the video. Every predetermined time
The recognition unit recognizes a user from the video imaged by the imaging unit,
The specifying unit specifies the arrangement pattern,
The setting unit corresponds to the role of the user with respect to the user based on the role of the first information corresponding to the user recognized by the recognition unit and the second information. The information processing apparatus according to claim 1, wherein the priority is set.   The information processing apparatus according to claim 1, wherein the imaging unit captures images in all directions to obtain the images in all directions. The information processing apparatus according to any one of claims 1 to 9,
A server device having the first information and the second information;
Conference system. An information processing method in an information processing apparatus,
An imaging step for imaging video;
An input step for inputting voice;
A recognition step for recognizing a user from the captured image;
A specific step of specifying an arrangement pattern indicating in which direction the recognized user is arranged with respect to the information processing device in the captured image;
An acquisition step of acquiring first information including at least information indicating a user and a role of the user associated with the information indicating the user;
Based on the role of the first information corresponding to the recognized user and the second information in which the role and the priority are associated in advance, the priority corresponding to the role of the user is given to the user. A setting step to set
When voice is input from a plurality of directions, the direction of the user having a role having a high priority among the directions of the users specified by the arrangement pattern among the plurality of directions is given priority, Cutting out a video region including the user corresponding to the direction from the video;
A transmission step of transmitting the cut out video area;
An information processing method comprising: In a computer provided with a video section for capturing video and an input section for inputting sound,
A recognition step of recognizing a user from the video imaged by the imaging unit;
A specifying step of specifying an arrangement pattern indicating in which direction the recognized user is arranged with respect to the information processing device in the video imaged by the imaging unit;
An acquisition step of acquiring first information including at least information indicating a user and a role of the user associated with the information indicating the user;
Based on the role of the first information corresponding to the user recognized by the recognition unit, and the second information in which the role and the priority are associated in advance, the user's role with respect to the user A setting step for setting the priority corresponding to
When voice is input from a plurality of directions by the input unit, priority is given to the direction of a user having a higher priority among the user directions specified by the arrangement pattern in the plurality of directions. Cutting out a video region including the user corresponding to the direction from the video;
A transmission step of transmitting the cut out video area;
A program for running

Images