JP2020057875A - Communication terminal, communication system, imaging apparatus, and imaging method - Google Patents

Abstract

To perform optimal exposure control for more participants.SOLUTION: The communication terminal for communicating with a plurality of terminals located at a first site and located at another site includes: an output unit for outputting a first site image, which is an image of the first site to the plurality of terminals; an acquisition unit for acquiring user's viewpoint information for the first site image displayed at the site of the terminal from each of the plurality of terminals; and an exposure control unit for using a region having a high degree of attention from the user as a photometric region in the first site image obtained from the plurality of pieces of viewpoint information for the first site image.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a communication terminal, a communication system, an imaging device, and an imaging method.

  2. Description of the Related Art Conventionally, there has been a communication system such as a video conference system for holding a conference between remote sites via a communication network such as the Internet. In such a video conference system, a video imaged by an imaging device at each site is transmitted and received between the sites.

  For example, there is an image pickup apparatus that suppresses blackout under backlight and overexposure during over-directed light (for example, see Patent Document 1). The imaging apparatus disclosed in Patent Document 1 determines a main subject area and a non-main subject area based on the magnitude of the time ratio in which the photographer's viewpoint is present in each area, and exposes the main subject area and the non-main subject area to each other. Perform control.

  In a video conferencing system in which participants are present at a plurality of locations, the images of the participants at the location taken by the imaging device at one location are viewed by the participants at the other locations. When the imaging device disclosed in Patent Literature 1 is used in such a video conference system, it is possible for one participant at another base to control the exposure of a subject in an area where the viewpoint is present. However, exposure control may be performed on a participant at a site that is included in the plurality of sites and is not the other site, in an area different from an area where the viewpoint of the participant exists.

  Therefore, a communication terminal, a communication system, an imaging device, and an imaging method according to the present disclosure aim to perform optimal exposure control for more participants.

  A communication terminal according to an embodiment of the present invention is a communication terminal that is located at a first base and communicates with a plurality of terminals located at other bases, and is a first base that is an image of the first base. An output unit that outputs an image to the plurality of terminals, an acquisition unit that obtains, from each of the plurality of terminals, viewpoint information of a user with respect to the first base image displayed at the base of the terminal, and the first base image And an exposure control unit that performs exposure control that sets a region of high interest of the user in the first base image obtained from the plurality of pieces of viewpoint information as a photometric region.

  A communication system according to an embodiment of the present invention is a communication system including a plurality of communication terminals arranged at a plurality of bases and communicating with each other, wherein each of the communication terminals is a first base where the communication terminal is arranged. A first output unit that outputs a first base image that is a captured image to each of the other communication terminals, and a base image acquired from each of the other communication terminals and displayed at the first base, A second output unit that outputs the first user's viewpoint information for the base image, which is an image of a base where the other communication terminals are arranged, to the other communication terminal, and from the other communication terminals, An acquisition unit for acquiring viewpoint information of the second user for the first site image displayed at the site where each of the communication terminals is arranged, and for the other communication terminal for the first site image And a exposure control unit that controls exposure to the attention of regions of high photometric area from the second user in said first base image obtained from the acquired plurality of viewpoint information from.

  An imaging device according to an embodiment of the present invention is an imaging device arranged at a first base, an imaging unit for acquiring a first base image which is an image obtained by imaging the first base, and an imaging unit arranged at another base. A communication terminal that communicates with the plurality of terminals, the communication terminal is a first output unit that outputs the first base image to the plurality of terminals, and the first terminal is obtained from each of the plurality of terminals and the first A second output unit that outputs, to the terminal, a base image displayed at the base, and viewpoint information of a first user with respect to the base image, which is an image of a base where the plurality of terminals are arranged; From each of the terminals, an acquisition unit that acquires viewpoint information of the second user for the first base image displayed at the base of the terminal, and a plurality of units obtained from the plurality of terminals for the first base image. Said viewpoint information And a exposure control unit that controls exposure to the region with a high level of interest from the second user in said first base image et resulting photometric area.

  An imaging method according to an embodiment of the present invention is an imaging method at a first base, a step of obtaining a first base image which is an image obtained by imaging the first base, and a plurality of bases arranged at other bases. Outputting the first site image to the plurality of terminals by communicating with a terminal, and a site image acquired from each of the plurality of terminals and displayed at the first site, wherein the plurality of terminals Outputting, to the terminal, viewpoint information of a first user with respect to a base image which is an image of a base where the respective bases are arranged; and the first base image displayed at the base of the terminal from each of the plurality of terminals. Obtaining viewpoint information of a second user with respect to the first base image obtained from the plurality of viewpoint information obtained from the plurality of terminals with respect to the first base image And performing exposure control for the region with a high level of interest from the second user in the image and metering area, said first base image after the exposure control, and outputting the plurality of terminals.

  According to the technology of the present disclosure, it is possible to perform optimal exposure control for more participants.

FIG. 1 is a diagram showing an example of a configuration of a video conference system according to Embodiment 1. FIG. 2 is a block diagram showing an example of a functional configuration of the server device according to the first embodiment. FIG. 2 is a block diagram showing an example of a functional configuration of the reservation server device according to the first embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the server device according to the first embodiment. FIG. 2 is a block diagram showing an example of a hardware configuration of a reservation server device according to the first embodiment. FIG. 2 is a block diagram showing an example of a functional configuration of the terminal system according to Embodiment 1. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the terminal device according to Embodiment 1. FIG. 2 is a plan view showing an example of the arrangement of participants and imaging units at a first base among the bases of the conference; The figure which shows an example of the image imaged by the imaging part of the 1st base of FIG. 8A. FIG. 7 is a diagram illustrating an example of a photometry area set for an image in exposure control. FIG. 8B is a diagram showing an example of application of the photometry area to the image of FIG. 8B. FIG. 9 is a diagram illustrating an example of weighting of a photometric area to the image in FIG. 9B in exposure control. The figure which shows an example of the gaze information of the participant of the 2nd base with respect to the image of the 1st base of FIG. 8B. The figure which shows an example of the gaze information of the participant of the 3rd base with respect to the image of the 1st base of FIG. 8B. The figure which shows an example of the image of FIG. 9B in which attention information of the photometry area was set using the line-of-sight information of the second base and the third base. Diagram showing an example of the direction of the speaker at the first base FIG. 9B is a diagram illustrating an example of the image of FIG. 9B in which attention information of a photometry area is set using a direction of a speaker at a first base. Diagram showing an example of attention information of each photometry area set from attention information of each base FIG. 13 is a diagram illustrating an example of attention information of each photometry area after weighting attention information of each photometry area in FIG. 13. Diagram showing an example in which regions of high interest are dispersed Flowchart showing an example of the operation of the terminal system according to Embodiment 1. FIG. 8 is a view for explaining a process of determining importance of attention information in the terminal system according to the second embodiment.

  2. Description of the Related Art In recent years, a video conference system for holding a conference between conference sites distant from each other via a communication network such as the Internet has become widespread. In such a conference system, at each site, images and sounds of participants and the like are captured and collected using a terminal system, and the images and sounds are converted into digital data and transmitted to terminal systems at other sites. You. At other sites, the transmitted image and sound are output by a display and a speaker. Thus, participants at a plurality of locations can hold a conference in a state close to a conference at the same location.

  However, in a conventional video conference system, an image captured by an imaging device is often subjected to optimal exposure control over an entire imaging range such as an entire conference room. The part that the participant is paying attention to is not necessarily controlled to obtain the optimal exposure. In particular, when an image of a place with a large difference in brightness is captured, a point that the participant wants to pay attention to may be overexposed, underexposed, or close to these points in the image on the transmission side. In such a case, a good image cannot be obtained even if the image quality is adjusted at the receiving site.

  For example, in a meeting room having a window on one wall, if the exposure is adjusted to the entire meeting room, the person on the window side will be backlit. For this reason, in the captured image, the image of the face of the person on the window side is in a state of blackening or close to it. On the other hand, the face of the person on the opposite side of the window is well lit by light, so that the image is brighter. In a scene where many participants pay attention to the facial expression of a person who is imaged brightly, even if image quality adjustment is performed at each site on the receiving side, a large correction is required from the brightness at the imaging location, so that good image quality is obtained. Coordination is difficult.

  This also applies to the imaging device of Patent Document 1. This imaging apparatus can perform exposure control on one of the participants at a plurality of locations viewing the images of the imaging apparatus, but performs exposure control on other participants. Can not do. Therefore, it is difficult to provide good images to many participants.

  Thus, the technology of the present disclosure provides a video conference terminal, a video conference system, an imaging device, and an imaging method that can perform optimal exposure control for more participants.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

(Embodiment 1)
2. Description of the Related Art A communication system is a system for transmitting and receiving information such as images and sounds between a plurality of sites via communication terminals arranged at a plurality of sites separated from each other. In the present embodiment, a description will be given assuming that the communication system is a video conference system 1 used for holding a conference between a plurality of sites. The video conference system 1 is an example of a communication system.

<Configuration of Video Conference System 1>
The video conference system 1 according to Embodiment 1 will be described. FIG. 1 is a diagram illustrating an example of a configuration of a video conference system 1 according to Embodiment 1. As shown in FIG. 1, a video conference system 1 according to the present embodiment includes a plurality of terminal devices 10, a server device 20, and a reservation server device 30. The plurality of terminal devices 10, the server device 20, and the reservation server device 30 are connected to each other via a network 40. The video conferencing system 1 is configured such that the terminal devices 10 arranged at a plurality of conference bases that are separated from each other transmit and receive information such as images and sounds between the plurality of bases via the network 40, so that Allows participants at bases to conduct meetings as if they were at the same location. In the present embodiment, the base of the conference is a conference room, but is not limited to this, and may be any location. Here, the terminal device 10 is an example of a communication terminal, and a participant is an example of a user.

  In the present embodiment, a description will be given assuming that four terminal devices 10a to 10d are arranged at four bases. However, two or more terminal devices may be arranged at two or more bases. In the following description, when the four terminal devices are individually specified and expressed, reference numerals “10a to 10d” may be used. When the four terminal devices are entirely or individually specified without being specified, the reference numerals are used. "10" may be used.

  The server device 20 controls communication between the terminal devices 10. For example, the server device 20 monitors a connection state as to whether each of the terminal devices 10 is connected to the server device 20. The server device 20 permits and denies participation of the terminal device 10 in the conference. For example, the server device 20 permits the terminal device 10 transmitting the authentication information to participate. The server device 20 calls each of the terminal devices 10 participating in the conference at the start of the conference. The server device 20 controls transmission and reception of information such as images and sounds to and from each terminal device 10 during the conference. The server device 20 acquires the information of the terminal devices 10 participating in the conference from the reservation server device 30, enables transmission and reception of information between the participating terminal devices 10, and allows the terminal device 10 to participate and the terminal device 10 not participating. Block transmission and reception of information between

  The reservation server device 30 manages the schedule of the meeting. The reservation server device 30 can be connected not only to the terminal device 10 but also to a computer device other than the terminal device 10 via the network 40. The reservation server device 30 receives and registers conference information from the terminal device 10 or another computer device. The meeting information includes information such as the date and time of the meeting, the location such as a base, the meeting participants, the agenda of the meeting, and the terminal device used. The reservation server device 30 issues authentication information for participating in a conference, such as a password. The terminal device 10 obtains information of a conference in which the terminal device 10 is to participate and authentication information for participation by making an inquiry to the reservation server device 30. Further, the reservation server device 30 transmits the meeting information and the authentication information to the server device 20, and causes the server device 20 to hold the meeting.

  The terminal device 10 communicates with the server device 20 and the reservation server device 30 via the network 40. The terminal device 10 obtains information of a conference in which the terminal device 10 is to participate and authentication information for participation by making an inquiry to the reservation server device 30. Upon receiving permission from the server device 20, the terminal device 10 participates in the conference and communicates with the other terminal devices 10 via the network 40 and the server device 20. The terminal device 10 acquires image and audio data of the participants of the conference at the base where the terminal device 10 is located, and transmits the data to the server device 20. The server device 20 transmits the acquired data to another terminal device 10 participating in the conference. Further, the terminal device 10 receives, from the server device 20, image and sound data acquired by another terminal device 10 and transmitted to the server device 20.

  For example, among the terminal devices 10a to 10d, when the terminal devices 10a to 10c participate in the conference and the terminal device 10d does not participate in the conference, the terminal devices 10a to 10d can be connected to the server device 20. The data transmitted by the terminal device 10a is transmitted to the terminal devices 10b and 10c via the server device 20, but is not transmitted to the terminal device 10d. Similarly, data transmitted from the terminal devices 10b and 10c is transmitted to the terminal devices 10a to 10c except for the terminal device 10b, but is not transmitted to the terminal device 10d. The data transmitted from the terminal device 10d is not transmitted to the terminal devices 10a to 10d.

  The network 40 is the Internet in the present embodiment, but is not limited to this. For example, the network 40 may be a LAN (Local Area Network), a WAN (Wide Area Network), a mobile communication network, a telephone line communication network, or another communication network. Network 40 may be a wireless network or a wired network.

  The functional configuration of the server device 20 and the reservation server device 30 will be described. FIG. 2 is a block diagram illustrating an example of a functional configuration of the server device 20 according to the first embodiment. FIG. 3 is a block diagram illustrating an example of a functional configuration of the reservation server device 30 according to the first embodiment.

  As shown in FIG. 2, the server device 20 includes a device control unit 20a, a communication unit 20b, a storage unit 20c, an operation unit 20d, and a display unit 20e. The communication unit 20b is connected to the network 40 and communicates with the terminal device 10 and the like via the network 40. The device control unit 20a controls the overall operation of the server device 20. The device control unit 20a transmits and receives information to and from the network 40 via the communication unit 20b. The storage unit 20c enables storage and retrieval of various information by the device control unit 20a. For example, the storage unit 20c may store image and audio data transmitted from each terminal device 10 during the conference. The operation unit 20d receives an operation, information, and a command input by an operator of the server device 20, and outputs the input to the device control unit 20a. The display unit 20e outputs the image data output from the device control unit 20a as an image. The display unit 20e may have a function of outputting audio data output from the device control unit 20a as audio.

  As illustrated in FIG. 3, the reservation server device 30 includes a device control unit 30a, a communication unit 30b, a storage unit 30c, an operation unit 30d, and a display unit 30e. The communication unit 30b is connected to the network 40 and communicates with the terminal device 10 and the like via the network 40. The device control unit 30a controls the entire operation of the reservation server device 30. The device control unit 30a transmits and receives information to and from the network 40 via the communication unit 30b. The storage unit 30c enables storage and retrieval of various types of information by the device control unit 30a. For example, the storage unit 30c may store conference information and authentication information. The operation unit 30d accepts an operation, information, and a command input by the operator of the reservation server device 30, and outputs the input to the device control unit 30a. The display unit 30e outputs the image data output from the device control unit 30a as an image. The display unit 30e may have a function of outputting audio data output from the device control unit 30a as audio.

  Although the server device 20 and the reservation server device 30 constitute separate and separate devices in the present embodiment, they may constitute an integrated device. Further, each device may be configured by one or more devices. When the device is configured by two or more devices, the two or more devices may be arranged in one device or may be separately arranged in two or more separated devices. In this specification and in the claims, the term “device” may mean not only one device, but also a system including a plurality of devices.

  The hardware configuration of the server device 20 and the reservation server device 30 will be described. FIG. 4 is a block diagram illustrating an example of a hardware configuration of the server device 20 according to the first embodiment. FIG. 5 is a block diagram illustrating an example of a hardware configuration of the reservation server device 30 according to the first embodiment.

  As shown in FIG. 4, the server device 20 includes a CPU (Central Processing Unit) 121, a nonvolatile storage device 122, a volatile storage device 123, a memory 124, a communication I / F (interface) 125, an operation An I / F 126 and a display device 127 are provided as constituent elements. The above components are connected to each other, for example, via a bus. The above components may be connected via any of wired communication and wireless communication.

  An example of the server device 20 is a computer device.

  The communication I / F 125 implements the function of the communication unit 20b. The communication I / F 125 may include a connection terminal, a communication circuit, and the like. The operation I / F 126 implements the function of the operation unit 20d. The operation I / F 126 may include input devices such as buttons, dials, keys, a touch panel, and a microphone for voice input. The display device 127 implements the function of the display unit 20e. The display device 127 may be a display such as a liquid crystal panel, an organic EL (Electroluminescence), an inorganic EL, and an electronic paper display. The display device 127 may be a touch panel that also serves as the operation I / F 126. The display device 127 may include a speaker.

  The memory 124 implements the function of the storage unit 20c. The memory 124 is configured by a storage device such as a volatile or nonvolatile semiconductor memory, a hard disk drive (HDD), or a solid state drive (SSD). Note that the memory 124 may include the nonvolatile storage device 122 and / or the volatile storage device 123.

  The CPU 121 implements the function of the device control unit 20a. The CPU 121 includes a processor and the like. An example of the non-volatile storage device 122 is a ROM (Read Only Memory), and an example of the volatile storage device 123 is a RAM (Random Access Memory). The program for operating the device control unit 20a is stored in the nonvolatile storage device 122 or the memory 124 in advance. The program is read out from the nonvolatile storage device 122 or the memory 124 to the volatile storage device 123 and expanded by the CPU 121. The CPU 121 executes each coded instruction in the program developed in the volatile storage device 123. The program may be stored on a recording medium such as a recording disk. In addition, the program may be transmitted via a wired network, a wireless network, broadcast, or the like, and may be loaded into the volatile storage device 123.

  The device control unit 20a may be realized by a program execution unit such as the CPU 121, may be realized by a circuit, or may be realized by a combination of a program execution unit and a circuit. For example, such components may be realized as an LSI (Large Scale Integration) which is an integrated circuit. Such components may be individually formed into one chip, or may be formed into one chip so as to include some or all of them. As the LSI, an FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, a reconfigurable processor that can reconfigure the connection and / or setting of circuit cells inside the LSI, or a plurality of specific applications. An ASIC (Application Specific Integrated Circuit) in which functional circuits are integrated into one may be used.

  As shown in FIG. 5, the reservation server device 30 includes a CPU 131, a nonvolatile storage device 132, a volatile storage device 133, a memory 134, a communication I / F 135, an operation I / F 136, and a display device 137. Is provided as a component. The above components are connected to each other, for example, via a bus. The above components may be connected via any of wired communication and wireless communication.

  An example of the reservation server device 30 is a computer device.

  The communication I / F 135 implements the function of the communication unit 30b. The configuration of communication I / F 135 is the same as the configuration described above for communication I / F 125. The operation I / F 136 implements the function of the operation unit 30d. The configuration of the operation I / F 136 is the same as the configuration described above for the operation I / F 126. The display device 137 implements the function of the display unit 30e. The configuration of the display device 137 is similar to the configuration of the display device 127 described above. The memory 134 implements the function of the storage unit 30c. The configuration of the memory 134 is the same as the configuration described above for the memory 124.

  The CPU 131 implements the function of the device control unit 30a. The CPU 131 includes a processor and the like. An example of the non-volatile storage device 132 is a ROM, and an example of the volatile storage device 133 is a RAM. The configurations of the CPU 131, the nonvolatile storage device 132, and the volatile storage device 133 are the same as those described above for the CPU 121, the nonvolatile storage device 122, and the volatile storage device 123.

  Next, a functional configuration of the terminal device 10 will be described. FIG. 6 is a block diagram showing an example of a functional configuration of the terminal system 100 according to Embodiment 1. The terminal device 10 constitutes a terminal system 100. The terminal system 100 is arranged at each base. The terminal system 100 includes a terminal device 10, an imaging unit 51, an audio input unit 52, an audio output unit 53, a display unit 54, and an input terminal at a base where the terminal system 100 is located (hereinafter, also referred to as “own base”). And a unit 55.

  The imaging unit 51 captures a still image and / or a moving image of a subject. An example of the imaging unit 51 is a camera that captures a digital image. The imaging unit 51 is connected to the terminal device 10 having the same own base via wired communication or wireless communication. The imaging unit 51 captures an image of a site such as a participant in a conference room at the own site, and outputs image data of the captured image to the terminal device 10.

  The audio input unit 52 acquires audio from the surroundings, converts the acquired audio into audio data such as an audio signal, and outputs the audio data. The sound input unit 52 outputs sound data according to the direction of the sound source. An example of the audio input unit 52 is a microphone array in which a plurality of microphones are arranged. The voice input unit 52 is connected to the terminal device 10 having the same base via wired communication or wireless communication. The voice input unit 52 acquires the voice of the participant or the like in the conference room at its own site, and outputs the voice data to the terminal device 10.

  The sound output unit 53 outputs sound to the surroundings. An example of the audio output unit 53 is a speaker. The audio output unit 53 is connected to the terminal device 10 at the same base via wired communication or wireless communication. The audio output unit 53 converts the audio data acquired from the terminal device 10 into audio, and outputs the audio to a participant or the like of the conference at the own location. For example, the audio output unit 53 outputs audio data transmitted from the terminal device 10 at another site.

  The display unit 54 outputs the input image data as an image. Examples of the display unit 54 are a liquid crystal panel, an organic EL, an inorganic EL, and an electronic paper display. The display unit 54 may be a touch panel that also has the function of the input unit 55. The display unit 54 is connected to the terminal device 10 at the same base via wired communication or wireless communication. The display unit 54 converts the image data acquired from the terminal device 10 into an image, and outputs the image to a participant of the conference at the own site. For example, the display unit 54 outputs image data transmitted from the terminal device 10 at another site. One display unit 54 may be arranged at one site, and may output images transmitted from the terminal devices 10 at all other sites. Alternatively, the same number of display units 54 as the other bases may be arranged, and each display unit 54 may output an image transmitted from the terminal device 10 at one of the other bases. Alternatively, the display unit 54 may be arranged for each participant at the base.

  The input unit 55 accepts an input by an operator of the base such as a participant of the conference and outputs input data such as a signal indicating the input information to the terminal device 10. The input unit 55 is connected to the terminal device 10 via wired communication or wireless communication. The input unit 55 receives an input for editing an image displayed by the display unit 54. Examples of the input unit 55 are a key, a mouse, and a touch panel.

  The terminal device 10 is connected to each component of the imaging unit 51, the audio input unit 52, the audio output unit 53, the display unit 54, and the input unit 55 existing at the own base via wired communication or wireless communication. It may be integrated with at least one of these components. Each component may be integrated with at least one of the other components. The wired communication or the wireless communication may be any communication such as a wired LAN or a wireless LAN.

  The terminal device 10 includes a first communication unit 11, a second communication unit 12, a control unit 13, a terminal operation unit 14, a terminal display unit 15, and a storage unit 16. The control unit 13 includes a line-of-sight estimation unit 13a, a sound direction estimation unit 13b, an attention information determination unit 13c, a synthesis unit 13d, an area determination unit 13e, and an exposure control unit 13f.

  The first communication unit 11 is connected to and communicates with the imaging unit 51, the audio input unit 52, the audio output unit 53, the display unit 54, and the input unit 55 of the own site. The second communication unit 12 is connected to the network 40, and communicates with the terminal device 10, the server device 20, and the reservation server device 30 at another base. Image data, audio data, and input signals output from the imaging unit 51, the audio input unit 52, and the input unit 55 of the own base are input to the control unit 13 via the first communication unit 11, and the processing of the control unit 13 is performed. After being received, it is transmitted to the terminal device 10 at another site via the second communication unit 12. In addition, the image data and the audio data transmitted from the terminal device 10 at the other base are input to the control unit 13 via the second communication unit 12, and after being processed by the control unit 13, the first communication unit 11 The data is output to the audio output unit 53 and the display unit 54 via the display unit.

  The terminal operation unit 14 receives an operation, information, and a command input by an operator of the terminal device 10 and outputs the input to the control unit 13. The terminal display unit 15 outputs data output from the control unit 13 as an image. Further, the terminal display unit 15 may have a function of outputting audio data output from the control unit 13 as audio.

  The storage unit 16 enables storage and retrieval of various information. For example, the storage unit 16 stores information on the relative and / or absolute positions of the imaging unit 51, the audio input unit 52, the audio output unit 53, and the display unit 54 of the own base, and the identification information of the base of the conference Etc. are stored in advance. The storage unit 16 is connected to the control unit 13.

  The control unit 13 controls the overall operation of the terminal device 10. The control unit 13 transmits and receives data to and from the imaging unit 51, the audio input unit 52, the audio output unit 53, the display unit 54, and the input unit 55 of the own base via the first communication unit 11. The control unit 13 transmits and receives data to and from the terminal device 10, the server device 20, and the reservation server device 30 at another site via the second communication unit 12 and the network 40.

  The line-of-sight estimating unit 13a estimates the line of sight of the participant at the own base. Specifically, the line-of-sight estimating unit 13a acquires image data captured by the image capturing unit 51 at its own site, and estimates the direction of the line-of-sight of the participant shown in the image of the image data. Extraction of a person from an image and estimation of the line of sight of the extracted person can be realized by a known technique.

  Further, the line-of-sight estimation unit 13 a acquires from the storage unit 16 information on the relative positions of the imaging unit 51 and the display unit 54 at the own base. The gaze estimation unit 13a estimates the position and range of the attention area, which is the area indicated by each gaze on the screen of the display unit 54, using the information on the estimated gaze direction and the relative position. The information on the relative positions of the imaging unit 51 and the display unit 54 includes the relative relationship between the position of the imaging unit 51 and the position of the display unit 54, and the imaging direction of the imaging unit 51 (also referred to as “optical axis direction”). And the relative relationship between the orientation of the screen of the display unit 54.

  Further, the gaze estimating unit 13a calculates pixel coordinates of the position and range of the attention area on the image displayed by the display unit 54, using the estimation result of the attention area. The pixel coordinates are two-dimensional coordinates set in the image, and one pixel, that is, one pixel is defined as one unit. Then, the line-of-sight estimation unit 13a outputs attention information including the pixel coordinates of the position and range of the attention area and identification information such as the ID of the own base in association with the attention information determination unit 13c.

  The gaze estimating unit 13a may acquire data of an image displayed by the display unit 54 of the own site, and extract a participant shown in the image. The eye gaze estimating unit 13a may specify a region of interest in which the extracted participant is projected by comparing the position and range of the extracted participant with the position and range of the region of interest on the image. . Then, the gaze estimation unit 13a sets identification information such as an ID for the extracted participant, associates the identification information with the pixel coordinates of the position and range of the attention area, and includes the identification information in the attention information. Good.

  The sound direction estimating unit 13b estimates the direction of the speaker among the participants at the base. Specifically, the sound direction estimating unit 13b acquires the sound data acquired by the sound input unit 52 of the own base, and estimates the direction of the sound source of the sound data. An example of the direction of the sound source is a direction from the sound input unit 52. The estimation of the direction of the sound source using the sound input unit 52 such as a microphone array can be realized by a known technique.

  Further, the sound direction estimating unit 13b acquires from the storage unit 16 information on the relative positions of the imaging unit 51 and the audio input unit 52 at the own base. The sound direction estimating unit 13b uses the information of the estimated direction of the sound source and the relative position to calculate the pixel coordinates of the position and range of the region of the sound source on the image captured by the imaging unit 51. presume. That is, the sound direction estimation unit 13b estimates the pixel coordinates of the position and range of the speaker on the image. The region of the sound source is the region of interest. Information on the relative positions of the imaging unit 51 and the audio input unit 52 includes the relative relationship between the position of the imaging unit 51 and the position of the audio input unit 52, the imaging direction of the imaging unit 51, and the collection of the audio input unit 52. And the relative relationship with the sound direction.

  Then, the sound direction estimation unit 13b outputs the attention information including the pixel coordinates of the position and range of the attention area and the identification information such as the ID of the own base to the attention information determination unit 13c. Note that the sound direction estimating unit 13b may acquire data of an image captured by the image capturing unit 51 of the own base, and extract a participant shown in the image. The sound direction estimating unit 13b compares the position and range of the extracted participant with the position and range of the region of interest on the image to specify the region of interest in which the extracted participant is projected. Good. Then, the sound direction estimating unit 13b sets identification information such as an ID for the extracted participant, associates the identification information with the pixel coordinates of the position and range of the attention area, and includes the identification information in the attention information. Is also good.

  The attention information determination unit 13c acquires attention information from the eye gaze estimation unit 13a and the sound direction estimation unit 13b. Further, the attention information determination unit 13c determines an index indicating the attention area on the image. Specifically, the attention information determination unit 13c generates a rectangular frame circumscribing the attention area on the image as an index, and calculates the pixel coordinates of the vertexes of the rectangular frame and the dimensions of the rectangular frame. The attention information determination unit 13c transmits the attention information including at least the coordinates and dimensions of the vertices of the rectangular frame and the identification information of the own site to the terminal device 10 at the other site via the second communication unit 12. . In addition, the attention information determination unit 13c outputs the attention information to the synthesis unit 13d.

  The combining unit 13d acquires the attention information from the attention information determination unit 13c of the own base, and acquires the attention information of the other base from each of the terminal devices at the other bases. Further, the combining unit 13d combines the attention information on the image captured by the imaging unit 51 of the own base by adding the attention information of the other base to the attention information of the own base. The attention information of the own base is the coordinates and dimensions of the vertex of the index indicating the attention area calculated by the sound direction estimation unit 13b of the own base, and is information based on the estimation result of the direction of the speaker. The attention information of the other base is the coordinates and the size of the vertex of the index indicating the attention area calculated by the gaze estimation unit 13a of the other base, and is information based on the estimation result of the gaze of the participant.

  The area determination unit 13e determines the area of interest by adding a predetermined weight to the attention information synthesized by the synthesis unit 13d. The area determining unit 13e outputs the information of the attention area to the exposure control unit 13f. Information on the weighting is stored in, for example, the storage unit 16.

  The exposure control unit 13f adjusts the exposure of the image captured by the imaging unit 51 of the own base using the information of the attention area. The exposure control unit 13 f transmits the image after the exposure adjustment to the terminal device 10 at another site via the second communication unit 12.

  Next, the hardware configuration of the terminal device 10 will be described. FIG. 7 is a block diagram showing an example of a hardware configuration of terminal device 10 according to Embodiment 1. As shown in FIG. 7, the terminal device 10 includes a CPU 111, a nonvolatile storage device 112, a volatile storage device 113, a first communication I / F 114, a second communication I / F 115, and an operation I / F 116. , A display device 117 and a memory 118 as constituent elements. The above components are connected to each other, for example, via a bus. The above components may be connected via any of wired communication and wireless communication.

  An example of the terminal device 10 is a computer device. The terminal system 100 including the terminal device 10 may be a system including a plurality of devices or a single device. In the case of one device, examples of the terminal system 100 are a computer device and a multifunctional television.

  The first communication I / F 114 implements the function of the first communication unit 11. The second communication I / F 115 implements the function of the second communication unit 12. The first communication I / F 114 and the second communication I / F 115 have the same configuration as that of the communication I / F 125 described above.

  The operation I / F 116 implements the function of the terminal operation unit 14. The configuration of the operation I / F 116 is the same as the configuration described above for the operation I / F 126. The display device 117 implements the function of the terminal display unit 15. The configuration of the display device 117 is the same as the configuration of the display device 127 described above. The memory 118 implements the function of the storage unit 16. The configuration of the memory 118 is similar to the configuration described above for the memory 124.

  The CPU 111 implements the function of each component of the control unit 13. The CPU 111 includes a processor and the like. An example of the non-volatile storage device 112 is a ROM, and an example of the volatile storage device 113 is a RAM. The configurations of the CPU 111, the nonvolatile storage device 112, and the volatile storage device 113 are the same as those described above for the CPU 121, the nonvolatile storage device 122, and the volatile storage device 123.

<Process of Terminal Device 10>
Details of the processing of the terminal device 10 will be described. In the following, the processing of the terminal device 10a of the first site CP1 of the four sites CP1 to CP4 will be described, but the same applies to the terminal devices 10b to 10d of the other sites CP2 to CP4, and therefore, the description thereof will be omitted. I do.

<Arrangement in the first base CP1>
First, the arrangement in the first base CP1 will be described. FIG. 8A is a plan view showing an example of the arrangement of the participants and the imaging unit 51 at the first site CP1 of the conference sites CP1 to CP4. FIG. 8B is a diagram illustrating an example of an image captured by the imaging unit 51 of the first base CP1 in FIG. 8A.

  As shown in FIG. 8A, at the first base CP1, one imaging unit 51 and three participants PA to PC are located around a rectangular conference desk MD. The imaging unit 51 and the participant PB are located facing each other, and the participant PA and the participant PC are located facing each other. The participants PA to PC are located within the field of view of the imaging unit 51. The imaging unit 51 is disposed above the display unit 54, and the imaging direction of the imaging unit 51 and the direction of the screen of the display unit 54 are substantially parallel. The directional direction of the screen is a direction perpendicular to the screen. Such an imaging unit 51 outputs an image I1 as shown in FIG. 8B by imaging the participants PA to PC. In the image I1, participant images PA1 to PC1 and a conference machine image MD1, which are images of the participants PA to PC and the conference machine MD, respectively, are displayed.

<Exposure control of exposure controller 13f>
Next, the exposure control of the exposure control unit 13f will be described. FIG. 9A is a diagram illustrating an example of a photometry area set for an image in exposure control. FIG. 9B is a diagram illustrating an application example of the photometry area to the image I1 of FIG. 8B. As shown in FIG. 9A, in exposure control, the image I is divided into a plurality of photometric areas Imn. The photometric area Imn is an area of a minimum unit for calculating a photometric value. In FIG. 9A, the plurality of photometric regions Imn are formed by dividing the image I into m parts in the horizontal direction and n parts in the vertical direction.

  For each photometric region Imn, a photometric value is calculated by integrating a luminance signal indicating a pixel value of a pixel included in the photometric region Imn. Further, a weight is assigned to each photometric region Imn. Then, an average value of the weighted photometric values obtained by weighting the photometric values of all the photometric regions Imn is calculated as an AE (acoustic emission :) evaluation value. That is, the weighted average of the photometric values of all the photometric regions Imn is calculated as the AE evaluation value. The difference between the AE evaluation value and the target luminance value is calculated as an error amount. Then, exposure control is performed by controlling the gain, the exposure time, and the like so that the error amount falls within a predetermined range.

  For example, when the weighting of all the photometric regions Imn is the same, the exposure is uniformly adjusted over the entire visual field of the image. For example, when performing exposure control with a specific gravity relative to the participant image PB1 shown in FIG. 9B, the weight of the photometric region Imn near the participant image PB1 is set to be larger than the weight of the photometric region Imn other than the participant image PB1. Therefore, by controlling the weighting for each photometric region Imn, it is possible to perform exposure control according to the target subject in the image. As a result, the occurrence of overexposure, underexposure, or a state close to these is suppressed in the target subject.

  For example, the exposure control unit 13f sets a weight for each photometric region Imn of the image I1 using the information of the weighted attention area acquired from the area determination unit 13e. FIG. 10 is a diagram illustrating an example of weighting the photometric area Imn to the image I1 of FIG. 9B in exposure control. In the image I1 of FIG. 10, the attention area is shown, and the attention area is the participant image PB1 and an area around the participant image PB1. Among the attention areas, the photometric area Imnc indicated by dark dots is an area that is receiving the most attention, that is, an area with the highest attention degree. The photometric area Imnb indicated by a thin dot is an area of interest next, that is, an area of next highest interest. The plain photometry area Imna is an area that is not noticed, that is, an area that is not noticed.

  For example, the exposure control unit 13f sets the weight of the exposure control on the photometric region Imnc to be the largest. The exposure control unit 13f sets the weight for the photometric region Imnb to be smaller than that of the photometric region Imnc, but larger than the normal weight set in advance for the image I1. The exposure control unit 13f sets the weight for the photometric area Imna to be smaller than the normal weight, or eliminates the weight, that is, sets the weight to zero. As described above, the exposure control unit 13f performs the exposure control with the specific gravity placed on the attention area by weighting the attention area more heavily.

<Estimation processing of attention area by gaze estimation section 13a>
The process of estimating the region of interest performed by the gaze estimation unit 13a will be described. FIG. 11A is a diagram illustrating an example of line-of-sight information of the participant Vb of the second site CP2 with respect to the image of the first site CP1 in FIG. 8B. FIG. 11B is a diagram illustrating an example of line-of-sight information of the participant Vc of the third site CP3 with respect to the image of the first site CP1 in FIG. 8B. FIG. 11C is a diagram illustrating an example of the image in FIG. 9B in which attention information of the photometry area is set using the line-of-sight information of the second site CP2 and the third site CP3.

  As shown in FIGS. 11A and 11B, the image I1 captured by the imaging unit 51 of the first location CP1 is transmitted by the terminal device 10a to the terminal devices 10b to 10d at other locations. For example, an image I2 similar to the image I1 is displayed on the display unit 54 of the second site CP2, and an image I3 similar to the image I1 is displayed on the display unit 54 of the third site CP3.

  As shown in FIG. 11A, the line-of-sight estimating unit 13a of the terminal device 10b of the second site CP2 estimates the direction of the line-of-sight of the participant Vb using the image of the participant Vb imaged by the imaging unit 51 of the own site. I do. Further, the gaze estimating unit 13a uses the direction of the gaze of the participant Vb and the relative position information of the imaging unit 51 and the display unit 54 of the second base CP2 to display the information on the display unit 54 of the own base. The pixel coordinates of the position and range of the attention area F2 on the image I2 to be processed are calculated.

  The attention information determination unit 13c of the terminal device 10b uses the position and the pixel coordinates of the attention area F2 to determine the pixel coordinates (x2, y2) of one vertex of the frame F2f circumscribing the attention area F2 and the dimensions of the frame F2f. (W2, z2) is calculated. The pixel coordinates of the vertices of the frame F2f and the dimensions of the frame are calculated using the pixel coordinates of the image I2. The component “x2” of the pixel coordinates is a horizontal component of the image I2, and the component “y2” is a vertical component of the image I2. The dimension component “w2” is the horizontal dimension of the image I2, and the component “z2” is the vertical dimension of the image I2. The horizontal direction and the vertical direction of the image I2 are the horizontal and vertical arrangement directions of the pixels arranged in a lattice, respectively.

  The attention information determination unit 13c determines the ID “CP2” of the own base, the ID “C” of the participant of the first base CP1 which is the target of the attention area, the pixel coordinates (x2, y2) of the frame vertex, and the frame size. Attention information including (w2, z2) in association with each other is transmitted to the terminal device 10a of the first base CP1.

  As illustrated in FIG. 11B, the line-of-sight estimation unit 13a of the terminal device 10c of the third site CP3 estimates the direction of the line-of-sight of the participant Vc using the image of the participant Vc captured by the imaging unit 51 of the own site. I do. Further, the gaze estimating unit 13a uses the direction of the gaze of the participant Vc and the information on the relative positions of the imaging unit 51 and the display unit 54 of the third base CP3 to generate an image on the display unit 54 of the own base. The pixel coordinates of the position and range of the attention area F3 on I3 are calculated.

  The attention information determination unit 13c of the terminal device 10c uses the position and the pixel coordinates of the area of the attention area F3 to determine the pixel coordinates (x3, y3) and the frame size (w3, z3) of the vertex of the circumscribed frame F3f of the attention area F3. Is calculated based on the pixel coordinates of the image I3.

  The attention information determination unit 13c determines the ID “CP3” of the own base, the ID “B” of the participant of the first base CP1 that is the target of the attention area, the pixel coordinates (x3, y3) of the frame vertex, and the frame size. Attention information including (w3, z3) in association with each other is transmitted to the terminal device 10a of the first base CP1.

  Further, the synthesizing unit 13d of the terminal device 10a at the first base CP1 generates an image I1A by performing a process of dividing a photometric area on the image I1. The combining unit 13d applies the attention information received from the terminal device 10 at each base to the image I1A.

  As illustrated in FIG. 11C, for example, the combining unit 13d forms a frame F2f on the image I1A by applying attention information of the terminal device 10b at the second base CP2. Further, the combining unit 13d forms the frame F3f on the image I1A by applying the attention information of the terminal device 10c at the third base CP3.

  The synthesizing unit 13d determines each of the photometry area including the entirety in the frame F2f and the photometry area including the entirety in the frame F3f as the first attention area which is the photometry area having the highest interest. The combining unit 13d determines each of the photometry area partially included in the frame F2f and the photometry area partially included in the frame F3f as a second attention area that is a photometry area with the next highest interest. The synthesizing unit 13d determines each of the photometry area not included in the frame F2f and the photometry area not included in the frame F3f as a non-attention area that is a photometry area that is not observed.

  Note that the photometric regions in the first region of interest may be further differentiated by extracting the participant images PA1 to PC1 in the image I1A. For example, the synthesizing unit 13d determines that the degree of interest of the photometric region including at least partially the participant images PA1 to PC1 is higher than the photometric region not including the participant images PA1 to PC1 in the first region of interest. You may.

  The combining unit 13d determines a region including the first region of interest and the second region of interest as a target photometric region. In FIG. 11C, the first region of interest is indicated by dark dots, the second region of interest is indicated by light dots, and the non-target region is solid. As described above, the terminal device 10a of the first base CP1 determines the target photometric region in the image I1 depicting the first base CP1, using the target region based on the line-of-sight information of the other base. Here, the information of the attention area based on the line-of-sight information is an example of viewpoint information.

<Estimation processing of attention area by sound direction estimation section 13b>
The processing of the attention area by the sound direction estimation unit 13b will be described. FIG. 12A is a diagram illustrating an example of the direction of the speaker at the first base CP1. FIG. 12B is a diagram illustrating an example of the image of FIG. 9B in which attention information of the photometry area is set using the direction of the speaker at the first base CP1.

  As shown in FIG. 12A, at the first base CP1, the voice input unit 52 is disposed below the imaging unit 51 and the display unit 54, and obtains the voices of the participants PA to PC. For example, when the participant PC speaks, the voice input unit 52 acquires the voice of the participant PC, and outputs the voice data to the sound direction estimation unit 13b of the terminal device 10a. The sound direction estimating unit 13b estimates the direction from the sound input unit 52 to the participant PC, which is the sound source of the sound data, using the obtained sound data. The sound direction estimating unit 13b uses the direction of the participant PC and information on the relative positions of the imaging unit 51 and the voice input unit 52 at the own location, and the attention area on the image I1 captured by the imaging unit 51. Then, the pixel coordinates of the position and range of the sound source area are calculated.

  The attention information determination unit 13c of the terminal device 10a calculates the pixel coordinates of one vertex of the frame FAf circumscribing the sound source region and the size of the frame FAf using the position and range pixel coordinates of the sound source region.

  The attention information determination unit 13c includes the ID “CP1” of its own base, the ID “C” of the participant who is the target of the attention area, the pixel coordinates of the vertex of the frame FAf, and the dimensions of the frame FAf in association with each other. The attention information is output to the combining unit 13d of the terminal device 10a.

  Further, the synthesizing unit 13d generates an image I1A by performing a process of dividing the photometry area on the image I1. The combining unit 13d applies the attention information obtained from the attention information determination unit 13c to the image I1A.

  As shown in FIG. 12B, for example, the combining unit 13d forms a frame FAf on the image I1A by applying the attention information. The synthesizing unit 13d determines the photometric region that is entirely included in the frame FAf as the first region of interest, determines the photometric region that is partially included in the frame FAf as the second region of interest, and determines the photometric region that is not included in the frame FAf. Is determined as a non-attention area.

  Note that the photometric regions in the first region of interest may be further differentiated by extracting the participant images PA1 to PC1 in the image I1A. For example, the synthesizing unit 13d sets the degree of attention of the photometry area including at least partially the participant images PA1 to PC1 in the first attention area higher than that of the photometry area not including the participant images PA1 to PC1. Is also good.

  The combining unit 13d determines a region including the first region of interest and the second region of interest as a target photometric region. As described above, the terminal device 10a of the first base CP1 determines the target photometric region in the image I1 that captures the first base CP1, using the target region based on the sound direction information of the local base.

<Attention area determination processing of combining section 13d>
The attention area determination processing of the combining unit 13d will be described. FIG. 13 is a diagram illustrating an example of attention information of each photometry area set from attention information of each base. FIG. 13 shows a method of determining the area of interest for the image I1A of the imaging unit 51 of the first base CP1. In FIG. 13, the speaker is the participant PA.

  The synthesizing unit 13d of the terminal device 10a of the first base CP1 acquires the information of the target photometric region based on the sound direction information from the target information determining unit 13c of the own base for the image I1A, and the terminal of the other base CP2 to CP4. From the attention information determination unit 13c of each of the devices 10b to 10d, information on the attention photometry area based on the line-of-sight information is acquired. The information on the target photometry area includes the positions and the degrees of interest of the target photometry areas and the non-target areas, that is, includes the degrees of interest of the respective photometry areas. Note that each attention information determination unit 13c quantifies the attention degree, and the greater the value, the higher the attention degree.

  In FIG. 13, for example, the attention degree of the first attention area is “3”, the attention degree of the second attention area is “2”, and the attention degree of the non-interest area is “0”. For example, in the photometric area of interest at the first base CP1, the photometric area including the face of the participant image PA1 is the first area of interest, and the photometric area around the participant image PA1 is the second area of interest. The photometric area of the photometric area of interest at the other bases CP2 to CP4 is a second area of interest.

  The synthesizing unit 13d adds the remarkability of the corresponding photometry areas of the other photometry areas of the other locations CP2 to CP4 to the remarkability of each photometry area of the photometry area of interest at the first location CP1. The remarkability of the photometry areas of the other locations CP2 to CP4 located at the same position on the image I1A as the photometry area is added to the remarkability of the photometry area of the first location CP1. Thus, the combining unit 13d generates addition attention information in which the attention degree of each photometric region is the attention degree after the addition, and outputs the information to the area determination unit 13e.

  The area determination unit 13e extracts a photometric region having the highest degree of interest from the acquired additional information of interest. For example, in FIG. 13, the degree of attention “9” is the largest, and the photometric region with the degree of attention “9” can be regarded as the most noticeable.

  For example, as shown in FIG. 14, the area determining unit 13 e resets the degree of interest in the surrounding photometric area around the photometric area with the degree of interest “9” according to a preset weighting method. Determine the area of interest. FIG. 14 is a diagram illustrating an example of the attention information of each photometry area after weighting the attention information of each photometry area in FIG. 13.

  In FIG. 14, the weighting method is an emphasis type in which the degree of attention is gradually reduced, for example, at a predetermined rate from the center of the photometric area having the degree of interest “9” toward the periphery. However, the weighting method is not limited to the method of FIG. For example, the weighting method may be a spot format in which the degree of interest is set only in the photometric region with the highest degree of interest or in a photometric region with a predetermined degree of interest or higher. The area determination unit 13e outputs attention area information including the attention degree of each photometry area after the setting to the exposure control unit 13f.

  The exposure control unit 13f performs exposure control on each photometric region of the image I1A with a weight corresponding to the degree of interest of each photometric region included in the attention area information. That is, the exposure control unit 13f uses the attention area information for weighting the photometry area in the exposure control. This makes it possible to perform optimal exposure control on an area where the most people are paying attention.

  Further, in the added attention information, there may be cases where two or more photometry areas having the highest attention degree are dispersed. For example, FIG. 15 is a diagram illustrating an example in which regions of high interest are dispersed. FIG. 15 shows an area of interest, in which there are two separate areas of interest each including a photometric area with the maximum degree of interest “9”. That is, it is shown that the participant's attention point is split into two. In such a case, the exposure control unit 13f may determine the exposure control method from the magnitude of the difference between the photometric values of the two photometric regions with the attention level “9”.

  For example, when the absolute value of the difference between the photometric values is within a predetermined range, optimal exposure control can be performed on the two areas of interest, regardless of the exposure control for any of the photometric regions. For this reason, the exposure control unit 13f performs exposure control in accordance with one of the photometric regions.

  When the absolute value of the difference between the photometric values exceeds a predetermined range, for example, it is considered that one photometric region corresponds to an image captured in backlight and the other photometric region corresponds to an image captured in normal light. Can be In this case, the exposure control unit 13f switches the exposure control method to a mode in which the dynamic range is widened. Examples of such modes include histogram photometry, WDR (Wide Dynamic Range) synthesis, HDR (High Dynamic Range) synthesis, and the like. By the above switching, it is possible to generate an image that is easy for the user to see even in two attention areas having a large contrast difference.

<Operation of Terminal Device 10>
The operation of the terminal device 10 will be described. FIG. 16 is a flowchart showing an example of the operation of the terminal system 100 according to Embodiment 1. In the following description, the operation of the terminal device 10a at the first base CP1 will be described. The operations of the terminal devices 10b to 10d at the other bases CP2 to CP4 are the same as those of the terminal device 10a, and the description thereof will be omitted.

  As shown in FIG. 16, when the conference starts, the terminal device 10a obtains the attention information of the own base by calculating the information of the sound direction of the own base by calculating (step S1). Furthermore, the terminal device 10a acquires the attention information of the other base based on the line-of-sight information of the other base from the terminal devices 10b to 10d of the other base (Step S2).

  Further, the terminal device 10a acquires the added attention information by adding the attention information of the other base to the attention information of the own base (step S3). If the number of photometric regions with the highest degree of interest is one in the added attention information (NO in step S4), the terminal device 10a proceeds to the process in step S5, and the number of photometric regions with the highest degree of interest is 2 If the number is equal to or greater than one (YES in step S4), the process proceeds to step S6.

  In step S5, the terminal device 10a determines an attention area centered on one photometry area having the highest attention degree, and generates information on the attention area. Next, the terminal device 10a performs the exposure control in which the information of the attention area is weighted as the weight of the photometry area (Step S10). Next, the terminal device 10a proceeds to the process of step S11.

  Further, in step S6, the terminal device 10a calculates a difference between the photometric values between the photometric regions having the highest degree of interest. If all of the absolute values of the difference are within the predetermined numerical range (YES in step S7), the terminal device 10a proceeds to the process of step S8, and any of the absolute values of the difference is out of the predetermined numerical range. In this case (NO in step S7), the process proceeds to step S9. If the number of photometric regions with the highest degree of interest is three or more, two or more differences are calculated.

  In step S8, the terminal device 10a determines an attention area centered on one of the photometric areas having the highest attention degree, and generates information on the attention area. Next, the terminal device 10a proceeds to step S10.

  In step S9, the terminal device 10a determines that the shooting scene has a wide dynamic range, and switches the exposure control to an exposure control method that widens the dynamic range. Next, the terminal device 10a proceeds to the process of step S11.

  In step S11, when the terminal device 10a receives the instruction to end the conference (YES in step S11), the terminal device 10a ends a series of processes, and when the instruction to end the conference is not received (NO in step S11), returns to step S1. .

<Effects>
The terminal device 10a according to the first embodiment as described above communicates with a plurality of terminal devices 10b to 10d arranged at the first site CP1 and arranged at the other sites CP2 to CP4. The terminal device 10a includes a second communication unit 12 as an output unit that outputs a first site image, which is an image of the first site CP1, to the plurality of terminal devices 10b to 10d, and a plurality of terminal devices 10b to 10d, respectively. A first communication unit 11 as an acquisition unit for acquiring viewpoint information of the user with respect to the first base images displayed at the bases CP2 to CP4 of the terminal devices 10b to 10d, and a plurality of pieces of viewpoint information for the first base images. And an exposure control unit 13f that performs exposure control in which a region of high user interest in the first base image obtained from is set as a photometric region.

  According to the above configuration, the terminal device 10a controls the exposure of the first base image using the viewpoint information on the first base image at another base. The terminal device 10a can present a clear area to many users by performing exposure control in the first base image, using a high-attention area obtained from a plurality of pieces of viewpoint information as a photometric area. That is, the terminal device 10a can perform optimal exposure control for more users.

  Further, terminal device 10a according to Embodiment 1 may include a sound direction estimating unit 13b. The sound direction estimating unit 13b functions as a direction estimating unit that estimates the direction of the sound source from the sound data collected for the first site CP1, and uses the direction of the sound source to obtain the position information of the sound source in the first site image. May function as a position estimating unit for estimating. The exposure control unit 13f may perform exposure control on the first base image, using a region of high interest in the first base image obtained from a plurality of pieces of viewpoint information and position information of sound sources as a photometric region.

  According to the above configuration, the viewpoint information and the sound source position information used by the exposure control unit 13f for the exposure control include the viewpoint information at another base and the sound source position information at the first base CP1. For example, in the first base image, the position of the sound source can be regarded as a position where the user of another base has a high degree of attention. The terminal device 10a performs optimal exposure control for a larger number of users by performing exposure control in the first base image, where a region of high interest in viewpoint information and sound source position information is a photometric region. Can be.

  Further, terminal device 10a according to Embodiment 1 may include a gaze estimation unit 13a. The gaze estimation unit 13a may estimate the gaze of the user at the first base CP1 from the first base image. Further, the line-of-sight estimating unit 13a functions as a viewpoint generating unit, and is a second base image acquired from the terminal devices 10b to 10d and displayed at the first base, using the line-of-sight information. The viewpoint information indicating the viewpoint of the user of the first base CP1 with respect to the second base image which is an image of the base of 10d may be generated. Furthermore, the second communication unit 12 may output the viewpoint information generated by the gaze estimation unit 13a to the terminal devices 10b to 10d.

  According to the above configuration, the terminal device 10a can output, to each of the other terminal devices 10b to 10d, the viewpoint information of the first site CP1 with respect to the image obtained by capturing the site of the terminal device 10b to 10d. Therefore, each of the terminal devices 10a to 10d can perform exposure control using the viewpoint information of another base.

  Further, in terminal device 10a according to Embodiment 1, when there are a plurality of maximum regions that are regions with the highest degree of attention, exposure control unit 13f determines that the difference in luminance value between the maximum regions in the first base image is a predetermined value. When the range is exceeded, exposure control may be performed so as to widen the dynamic range. The brightness value may be a photometric value in a photometric area.

  According to the above configuration, when the difference in the luminance value between the maximum regions is outside the predetermined range, for example, one of the maximum regions corresponds to an image captured with backlight, and the other maximum region is captured with normal light. It can be considered to correspond to an image. By performing exposure control for expanding the dynamic range on such a maximum area, it is possible to generate an image that is easy for the user to see even in two areas having a large difference in brightness.

  The video conference system 1 as a communication system according to the first embodiment includes a plurality of terminal devices 10 arranged at a plurality of bases and communicating with each other. The terminal devices 10 each include a second communication unit 12 as a first output unit that outputs a first site image, which is an image of the first site where the terminal device 10 is arranged, to each of the other terminal devices 10, Is a base image acquired from each of the terminal devices 10 and displayed at the first base, and the viewpoint information of the first user with respect to the base image which is an image of the base where the other terminal devices 10 are respectively arranged. The attention information determination unit 13c as a second output unit that outputs to the terminal device 10, and the second terminal image from the other terminal device 10 for the first site image displayed at the site where each of the other terminal devices 10 is arranged. A first communication unit 11 as an acquisition unit for acquiring viewpoint information of the user, and a second base image in the first base image obtained from a plurality of viewpoint information obtained from other terminal devices 10 for the first base image. And a exposure control unit 13f that controls exposure to the region with a high level of interest from over The photometric area. According to the video conference system 1, the same effect as the terminal device 10 can be obtained.

  Further, the present invention may be an imaging device. For example, the imaging device of the present invention is arranged at the first base CP1. The imaging device includes an imaging unit 51 that acquires a first site image that is an image of the first site, and a terminal device 10a that communicates with a plurality of terminal devices 10b to 10d arranged at other sites. The terminal device 10a is a second communication unit 12 that outputs the first base image to the plurality of terminal devices 10b to 10d, and a base image acquired from each of the plurality of terminal devices 10b to 10d and displayed at the first base. An attention information determination unit 13c that outputs, to the terminal devices 10b to 10d, viewpoint information of a first user with respect to a site image which is an image of a site where the plurality of terminal devices 10b to 10d are arranged; A first communication unit 11 for acquiring viewpoint information of a second user for a first site image displayed at a site of the terminal device 10b to 10d from each of the terminal devices 10b to 10d; Exposure for performing exposure control in which a region of high interest from the second user in the first site image obtained from a plurality of pieces of viewpoint information obtained from 10b to 10d is a photometric region. And a control unit 13f. According to this imaging device, the same effect as that of the terminal device 10 can be obtained. Note that an example of the imaging device may be the terminal system 100.

  Further, the present invention may be an imaging method. For example, an imaging method according to the present invention is an imaging method at a first site, wherein a step of obtaining a first site image which is an image of the first site is performed, and a plurality of terminals arranged at other sites. By communicating with the step of outputting the first base image to the plurality of terminals, and a base image acquired from each of the plurality of terminals and displayed at the first base, wherein each of the plurality of terminals Outputting, to the terminal, viewpoint information of a first user with respect to a base image which is an image of a base where the base is arranged; and from each of the plurality of terminals, the first base image displayed at the base of the terminal. A step of obtaining viewpoint information of a second user; and, for the first base image, the first base image obtained from the plurality of viewpoint information obtained from the plurality of terminals. And performing exposure control of the degree of attention regions of high photometric area from the second user of the first base image after the exposure control, and outputting the plurality of terminals. According to this imaging method, effects similar to those of the above-described imaging device can be obtained. Such an imaging method may be realized by a circuit such as a CPU and an LSI, an IC card, a single module, or the like.

(Embodiment 2)
The terminal device according to Embodiment 2 weights attention information acquired from terminal devices at other bases and adds the attention information. Hereinafter, the second embodiment will be described focusing on the points different from the first embodiment, and the description of the same points as the first embodiment will be appropriately omitted.

  FIG. 17 is a diagram illustrating a process of determining the importance of attention information in the terminal system according to Embodiment 2. As shown in FIG. 17, at the first base CP1, participants PA to PC, an imaging unit 51a, and a display unit 54a exist. At the second base CP2, participants PD to PF, an imaging unit 51b, and a display unit 54b are present. The third base CP3 includes participants PG to PI, an imaging unit 51c, and a display unit 54c. The fourth base CP4 includes participants PJ to PL, an imaging unit 51d, and a display unit 54d.

  A case will be described in which the terminal device 10a at the first site CP1 controls the exposure of the image captured by the imaging unit 51a and transmits the image to the terminal devices 10b to 10d at the other sites CP2 to CP4. In FIG. 17, the participant PD at the second base CP2 is the last speaker, that is, the speaker who spoke most recently, and the participant PH at the third base CP3 is the speaker who spoke immediately before the participant PD.

  Each of the terminal devices 10a to 10d calculates attention information based on the information on the sound direction at its own base. Thereby, each of the terminal devices 10a to 10d can detect the uttering participant and the timing of the utterance at the own base. Then, each of the terminal devices 10a to 10d transmits the speaker information including the identification information of the participant who uttered at the own site and the utterance time in association with the terminal devices 10a to 10d at the other sites. Further, each of the terminal devices 10a to 10d accumulates the utterance histories of all the participants at the bases CP1 to CP4, and stores the speaker history information, which is the information of the histories of the bases of the uttered participants, in the storage unit 16. Hold.

  In addition, the attention information based on the information on the sound direction at the own base can include time information when the sound data acquired by the sound input unit 52 includes the sound detection time. The attention information based on the line-of-sight information at another site can include time information when the image data captured by the imaging unit 51 includes the imaging time.

  The terminal devices 10a to 10d respectively link the speaker history information of the storage unit 16, the attention information based on the sound direction information at the own base, and the attention information based on the line-of-sight information at the other base, for example, based on time. Attach. Then, each of the terminal devices 10a to 10d weights the attention information using the speaker history information.

  For example, the terminal device 10a sets the weight of attention information acquired from the terminal device 10b of the second base CP2 where the participant PD who is the last speaker is the largest to “importance A”, and sets the talk immediately before the participant PD. The weight of the attention information acquired from the terminal device 10b of the third base CP3 in which the participant PH who is the participant is located is set to the second largest “importance B”. Furthermore, the terminal device 10a sets the weight of the attention information generated and acquired by the terminal devices 10 at the bases other than the second base CP2 and the third base CP3 to the “importance C”.

  The terminal device 10a weights the attention information generated at its own base and the attention information obtained from another base by importance according to the base, and uses the weighted information when calculating the additional attention information. For example, the weighting may be a multiplication of the attention degree by a weight. As a result, it is possible to obtain an image in which the region of interest at the base where the speaker currently exists is easy to see.

  In addition, each of the terminal devices 10a to 10d may handle attention information generated by a terminal device at a base other than the base where the last speaker is located as invalid attention information. As a result, it is possible to obtain a video image whose exposure is optimally controlled only in a region of interest at the base where the speaker currently exists.

  Other configurations and operations of the terminal device according to the second embodiment are the same as those of the first embodiment, and a description thereof will not be repeated. According to the terminal device according to the second embodiment as described above, the same effect as in the first embodiment can be obtained.

  Further, the terminal device 10a according to the second embodiment acquires, from each of the plurality of terminal devices 10b to 10d, history information of audio data collected for the bases CP2 to CP4 of the terminal devices 10b to 10d, The control unit 13f uses the weighted viewpoint information obtained by weighting the viewpoint information acquired from the plurality of terminal devices 10b to 10d according to the history information of the audio data of each of the plurality of terminal devices 10b to 10d for exposure control. May be used.

  According to the above configuration, the attention level obtained from the weighted viewpoint information reflects the history information of the audio data. For example, by increasing the weight for the viewpoint information of the base whose acquisition time of the audio data is close to the present, the degree of attention obtained from the viewpoint information increases. For example, it can be considered that the closer the acquisition time of the audio data is from the present time, the higher the attention degree of the user of the other base is. Therefore, the terminal device 10a can perform optimal exposure control for more users.

(Other embodiments)
As mentioned above, although the example of the embodiment of the present invention was explained, the present invention is not limited to the above-mentioned embodiment and the modification. That is, various modifications and improvements are possible within the scope of the present invention. For example, various modifications made to the embodiment or the modified example, and forms constructed by combining components in different embodiments and modified examples are also included in the scope of the present invention.

  For example, in the terminal system 100 according to the embodiment, the imaging unit 51 has a function of imaging a participant at a base in order to obtain line-of-sight information and a function of imaging an image subjected to exposure control by the exposure control unit 13f. It also served, but is not limited to this. For example, two imaging units that realize each function may be provided.

  Further, the numbers such as the ordinal numbers and the quantities used above are all examples for specifically explaining the technology of the present invention, and the present invention is not limited to the illustrated numbers. Further, the connection relation between the constituent elements is illustrated for specifically explaining the technology of the present invention, and the connection relation for realizing the function of the present invention is not limited to this.

  The division of blocks in the functional block diagram is merely an example, and a plurality of blocks may be implemented as one block, one block may be divided into a plurality of blocks, and / or some functions may be moved to another block. Good. Also, a single piece of hardware or software may process functions of a plurality of blocks having similar functions in parallel or in a time-division manner.

1 Video conference system (communication system)
10, 10a, 10b, 10c, 10d Terminal device (communication terminal)
11 First communication unit (acquisition unit)
12 Second communication unit (output unit, first output unit)
13 control unit 13a gaze estimation unit (viewpoint generation unit)
13b Sound direction estimation unit (direction estimation unit, position estimation unit)
13c Attention information determination unit (second output unit)
13d synthesis unit 13e area determination unit 13f exposure control unit 51, 51a, 51b, 51c, 51d imaging unit 52 audio input unit 54, 54a, 54b, 54c, 54d display unit 100 terminal system (imaging device)

JP 05-227469 A

Claims (9)

A communication terminal that is located at the first base and communicates with a plurality of terminals located at other bases,
An output unit that outputs a first base image, which is an image of the first base, to the plurality of terminals,
From each of the plurality of terminals, an acquisition unit that acquires user viewpoint information for the first base image displayed at the base of the terminal,
A communication terminal comprising: an exposure control unit configured to perform an exposure control on the first base image as a photometric area in the first base image obtained from the plurality of pieces of viewpoint information and having a high degree of attention from the user in the first base image. . A direction estimating unit that estimates the direction of a sound source from audio data collected for the first base,
Using a direction of the sound source, further comprising a position estimating unit for estimating position information of the sound source in the first site image,
The exposure control unit performs an exposure control on the first base image, in which a region of high interest in the first base image obtained from the plurality of pieces of viewpoint information and the position information of the sound source is the photometric region. The communication terminal according to claim 1. A gaze estimating unit that estimates the gaze of the user at the first base from the first base image;
Using the line-of-sight information, the second base image acquired from the terminal and displayed at the first base, the first base image of the second base image is an image of the base of the terminal A viewpoint generation unit that generates the viewpoint information indicating the viewpoint of the user,
The communication terminal according to claim 1, wherein the output unit outputs the viewpoint information generated by the viewpoint generation unit to the terminal. The exposure control unit, when there are a plurality of maximum regions that are regions with the highest degree of attention, expands a dynamic range when a difference in luminance value between the maximum regions in the first base image exceeds a predetermined range. The communication terminal according to any one of claims 1 to 3, wherein the communication terminal performs exposure control. The acquisition unit acquires, from each of the plurality of terminals, history information of audio data collected for the base of the terminal,
The exposure control unit uses, for exposure control, weighted viewpoint information obtained by weighting the viewpoint information acquired from the plurality of terminals according to history information of the audio data of each of the plurality of terminals. The communication terminal according to any one of claims 1 to 4. A communication system including a plurality of communication terminals arranged at a plurality of bases and communicating with each other,
Each of the communication terminals,
A first output unit that outputs a first base image, which is an image of the first base where the communication terminal is arranged, to each of the other communication terminals,
First user viewpoint information for a base image obtained from each of the other communication terminals and displayed at the first base, the base image being an image of a base where each of the other communication terminals is arranged. A second output unit that outputs to the other communication terminal,
From each of the other communication terminals, an acquisition unit that acquires viewpoint information of the second user for the first site image displayed at the site where the other communication terminals are respectively arranged,
Exposure of the first base image to a photometric region, where a region of high interest from the second user in the first base image obtained from the plurality of pieces of viewpoint information obtained from the other communication terminals is a photometric region. A communication system comprising: an exposure control unit that performs control. An imaging device arranged at a first base,
An imaging unit that acquires a first base image that is an image of the first base,
A communication terminal that communicates with a plurality of terminals located at other bases,
The communication terminal,
A first output unit that outputs the first base image to the plurality of terminals,
A base image acquired from each of the plurality of terminals and displayed at the first base, wherein the viewpoint information of the first user with respect to the base image, which is an image of a base where the plurality of terminals are arranged, is obtained. A second output unit for outputting to the terminal,
From each of the plurality of terminals, an acquisition unit that acquires viewpoint information of the second user for the first base image displayed at the base of the terminal,
Exposure control for the first site image, where a region of high interest from the second user in the first site image obtained from the plurality of viewpoint information obtained from the plurality of terminals is a photometric region. An imaging apparatus comprising: an exposure control unit configured to perform the operation. The imaging unit,
A first imaging unit that captures an image for acquiring the viewpoint information,
The imaging device according to claim 7, further comprising: a second imaging unit configured to capture an image on which exposure control is performed by the exposure control unit. An imaging method at a first base,
Obtaining a first base image, which is an image of the first base,
By communicating with a plurality of terminals located at other bases, outputting the first base image to the plurality of terminals,
A base image acquired from each of the plurality of terminals and displayed at the first base, wherein the viewpoint information of the first user with respect to the base image, which is an image of a base where the plurality of terminals are arranged, is obtained. Outputting to a terminal;
From each of the plurality of terminals, obtaining the viewpoint information of the second user for the first base image displayed at the base of the terminal,
Exposure control for the first site image, where a region of high interest from the second user in the first site image obtained from the plurality of viewpoint information obtained from the plurality of terminals is a photometric region. Performing
Outputting the first base image after the exposure control to the plurality of terminals.

Images