JP2013016929A - Imaging apparatus, imaging method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus, an imaging method, and a program capable of collecting voice to be uttered by a person even when the person is not included within an imaging range.SOLUTION: When a conference terminal 1 for imaging participants 53-55 is panned to allow an imaging direction to be A3 so that the participants are not included within a camera imaging range B1, the participants 53-55 do not appear in an image P3 and the faces of the persons are not detected. In this case, an orientation direction of an array microphone is set to C3, and its voice collection range is set to D3 being an area excluding an area to be specified by the imaging direction A3 and the imaging range B1 from all the 360-degree directions with the conference terminal 1 as a center. Consequently, the area with the participants 53-55 reliably becomes the voice collection object area, and also the voice collection from an area where the absence of the participants 53-55 is determined is avoided. Thus, the voice uttered by the participants 53-55 can be reliably and clearly collected.

Description

  The present invention relates to an imaging apparatus, an imaging method, and a program in which an imaging unit and a sound collection unit are integrally configured.

  2. Description of the Related Art An imaging device is known in which a camera that captures an image and a microphone that collects sound (hereinafter abbreviated as “microphone”) are configured integrally with a housing. For example, a conference terminal device used for a remote conference captures an image of its own site using an imaging device, collects sound, and transmits images and audio to / from terminal devices at other sites via a network. Send and receive data.

  In such an imaging apparatus, a microphone that uses a unidirectional microphone is known as a microphone for collecting sound in order to reliably and clearly pick up the voice of a speaker in a conference (see, for example, Patent Document 1). .) The imaging device (microphone with camera) described in Patent Document 1 has a configuration in which the angle of view of the camera is substantially equal to the range of unidirectionality of the microphone. And if the face image is not recognized in the image captured by the camera, the voice is not taken in by the microphone so that the unnecessary voice is not taken in unless the speaker is shown. .

  In addition, a microphone using a known array microphone is known as a microphone for collecting sound of the imaging apparatus (see, for example, Patent Document 2). An array microphone is one in which a plurality of non-directional microphones are arranged in an array, and directivity in an arbitrary direction can be obtained by electrical control. The image pickup apparatus (video camera with a built-in microphone) described in Patent Document 2 using such an array microphone links the directivity characteristics of the array microphone with the camera shake angle and zoom angle. As a result, when the camera is pointed in the direction of the speaker, the array microphone is directed toward the speaker, and when the speaker is zoomed, the array microphone is directed sharply toward the speaker. Can be picked up effectively.

JP 2009-49734 A JP-A-10-155107

  However, in the inventions described in Patent Documents 1 and 2, a speaker is projected on a camera, and the direction of the microphone is determined based on the image and orientation of the camera. For this reason, in the case where an object other than the speaker or another participant is displayed on the camera, for example, when the speaker explains while displaying the whiteboard with the camera, in Japanese Patent Application Laid-Open No. 2004-151867, the voice capturing by the microphone is blocked. There was a problem that. Moreover, in patent document 2, since the direction of the array microphone linked to the camera is directed to the whiteboard to which the camera is directed, there is a problem that the voice of the speaker cannot be clearly captured.

  The present invention has been made to solve the above-described problems, and provides an imaging apparatus, an imaging method, and a program capable of collecting sound produced by a person even when the person is not included in the imaging range. For the purpose.

  According to the first aspect of the present invention, based on an image capturing unit that captures an image, a plurality of sound collecting units configured to be integrated with the image capturing unit and collecting sound, and an image capturing range of the image capturing unit. Determining whether or not a person is included in the imaging range, based on a control unit that controls a directivity direction and a sound collection range in which sound is collected by the plurality of sound collection units, and an image captured by the imaging unit A first determination unit that performs the sound collection when the first determination unit determines that a person is not included in the imaging range. An image pickup apparatus that controls the directivity direction and the sound collection range of the sound collection means so that at least a part of the area outside the image pickup range of the image pickup means is a sound collection target area of the sound collection means. Is provided.

  According to the first aspect, if a person is not included in the imaging range of the imaging means, at least a part of the area outside the imaging range can be set as a sound collection target area. It can be included in the range, and the voice uttered by the person can be reliably collected. In addition, since the area within the imaging range that does not include a person is excluded from the sound collection target area, the sound is not collected even if there is noise or the like having a generation source in that area. Sound can be collected more clearly.

  The imaging apparatus according to the first aspect may further include second determination means for determining whether or not the imaging range of the imaging means has changed. In this case, when the second determination unit determines that the imaging range has changed, the control unit determines the directing direction and the sound collection range of the sound collection unit based on the content of the change in the imaging range. You may control.

  When the imaging range changes, the person is expected to be in the imaging range before the change. Therefore, if the control means controls the directivity direction and the sound collection range of the sound collection means based on the contents of the change in the imaging range, the area where the person is present is surely included in the area to be collected. Can do. Therefore, it is possible to reliably and more clearly collect the voice uttered by the person.

  In the first aspect, the control means is determined by the second determination means that the imaging range has changed, and the change in the imaging range is caused by a change in the imaging direction of the imaging means. , When it is determined that the imaging direction after the change from the imaging direction before the change cannot be specified, all of the areas outside the imaging range of the imaging means among the areas where the sound collecting means can collect sound The direction of sound collection and the sound collection range of the sound collecting means may be controlled so that this area becomes a sound collection target area of the sound collecting means.

  When the imaging range changes, the person is expected to be in the imaging range before the change, but if the imaging direction after the change cannot be specified from before the change, based on the imaging direction after the change, The directivity direction and sound collection range before the change cannot be specified. Therefore, the control means controls the directivity direction and the sound collection range of the sound collection means and collects sound from all areas outside the image pickup range of the image pickup means among the areas that can be collected. The sound collected by the person can be collected more reliably and clearly. can do.

  In the first aspect, the control means determines that the imaging range has changed by the second determination means, and the change in the imaging range is caused by a change in the imaging direction of the imaging means. Then, when it is determined that the imaging direction after the change can be specified from the imaging direction before the change, the sound is collected from the sound collection target area of the sound collection means before the change of the imaging range. As described above, the directing direction and sound collection range of the sound collecting means may be controlled.

  When the imaging range changes, the person is expected to be in the imaging range before the change, and if the imaging direction after the change can be specified from before the change, the person changes based on the imaging direction after the change. The previous directivity direction and sound collection range can be specified. Therefore, the control means controls the directivity direction and the sound collection range of the sound collection means, and collects the sound from the sound collection target area of the sound collection means before the change of the imaging range. It is possible to collect sound from a person reliably and more clearly while avoiding collecting from an area where there is no person while reliably making the area to be collected.

  In the first aspect, when the control unit determines that the imaging range has changed and the change of the imaging range is caused by a change in the angle of view of the imaging unit by the second determination unit. In addition, a region obtained by excluding a region that overlaps the imaging range after the change in the angle of view from a region to be collected by the sound collection unit before the change in the angle of view is a sound collection target of the sound by the sound collection unit The directing direction and the sound collection range of the sound collecting means may be controlled so as to be in the region.

  When the imaging range changes, if the change is caused by a change in the angle of view, the person is expected to be in an area excluding the imaging range after the change from the imaging range before the change. Therefore, the control means controls the directivity direction and the sound collection range of the sound collection means so that the area overlapping the imaging range after the change in the angle of view is changed from the sound collection target area of the sound collection means before the change in the angle of view. By collecting the sound from the excluded area, the area where the person is present is surely the area to be collected, while collecting the sound from the area where the person is not present, and the sound emitted by the person is collected more reliably and clearly. Can sound.

  In the first aspect, the first determination unit recognizes a part having a human facial feature from the image captured by the imaging unit, and the imaging unit is configured to detect the part when the recognized part has a size larger than a predetermined size. It may be determined that a person is included in the range.

  If a part having a human face characteristic within a captured image that is less than or equal to a predetermined size is not detected as a person, a person who is not an imaging target by chance is included in the imaging range. Even if it is, the person does not become a control condition of the sound collecting means. Thereby, it is possible to prevent the control means from controlling in the wrong directivity direction and sound collection range, and it is possible to reliably collect the sound emitted by the person to be collected.

  According to the second aspect of the present invention, there is provided an imaging method executed in a computer in order to cause an imaging device in which an imaging unit that captures an image and a plurality of sound collection units that collect sound to function together. And a control step for controlling a directivity direction and a sound collection range for collecting sound by the plurality of sound collection means based on an image pickup range of the image by the image pickup means, and based on an image taken by the image pickup means, A first determination step of determining whether or not a person is included in the imaging range, and when it is determined in the first determination step that a person is not included in the imaging range, In the control step, among the areas where the sound collecting means can collect sound, at least a part of the area outside the imaging range of the imaging means is a target area for collecting sound by the sound collecting means. An imaging method orientation and sound collecting range of the sound collecting means is controlled is provided.

  According to the third aspect of the present invention, there is provided a program for causing an image pickup apparatus configured to integrate an image pickup means for picking up an image and a plurality of sound collection means for collecting sound, Based on the imaging range of the image by the imaging means, a control step for controlling the directivity direction and the sound collection range for collecting sound by the plurality of sound collection means, and the imaging range based on the image captured by the imaging means A first determination step for determining whether or not a person is included therein, and when the first determination step determines that a person is not included in the imaging range, the control step The sound collection means collects the sound so that at least a part of the area outside the imaging range of the image pickup means is a sound collection target area of the sound collection means. Program pointing direction and sound collecting range means is controlled is provided.

  By executing the processing according to the imaging method according to the second aspect on the computer of the imaging apparatus, or by executing the program according to the third aspect and causing the computer to function as the imaging apparatus, the same effects as in the first aspect are obtained. Can be obtained.

It is a perspective view of the conference terminal 1 and PC9. 2 is a block diagram showing an electrical configuration of the conference terminal 1. FIG. 4 is a flowchart of a program executed on the conference terminal 1. It is a figure which shows the directional direction C1 and the sound collection range D1 set according to the imaging direction A1, the imaging range B1, etc. of the conference terminal 1. It is a figure which shows the directional direction C1 and the sound collection range D1 set according to the imaging direction A2, the imaging range B1, etc. of the conference terminal 1. It is a figure which shows the directivity direction C3 set according to the imaging direction A3, the imaging range B1, etc. of the conference terminal 1, and the sound collection range D3. It is a figure which shows the directivity direction C1 and the sound collection range D4 set according to the imaging direction A1, the imaging range B4, etc. of the conference terminal 1. It is a figure which shows the directional direction C1 and the sound collection range D1 set according to the imaging direction A1, the imaging range B5, etc. of the conference terminal 1. It is a figure which shows the directivity direction C6 set according to the imaging direction A6 of the conference terminal 1, the imaging range B1, etc., and the sound collection range D6. It is a figure which shows the sound collection range D7 set according to the imaging direction A7 of the conference terminal 1, imaging range B1, etc. FIG.

  Hereinafter, a conference terminal 1 which is an embodiment of an imaging apparatus according to the present invention will be described with reference to the drawings. Note that the drawings to be referred to are used to explain technical features that can be adopted by the present invention, and the configuration of the apparatus described, flowcharts of various processes, and the like are merely illustrative examples.

  First, the schematic configuration of the conference terminal 1 will be described with reference to FIG. The conference terminal 1 shown in FIG. 1 includes an array microphone 25, a speaker 27, a camera 28, and an operation unit 29. The conference terminal 1 is a device that can capture an image with the camera 28, collect sound with the array microphone 25, and generate sound with the speaker 27. The conference terminal 1 includes a rotation shaft 3 at the upper end of the housing 4 and is configured to rotate a part of the housing 4 around the rotation shaft 3 so that the lower end side can be opened and closed. By opening the lower end side of the housing 4, the user can set the posture of the housing 4 to be independent, that is, the posture in use (see FIG. 1). Further, by closing the lower end side of the housing 4, the housing 4 can be folded, that is, not in use (not shown).

  The conference terminal 1 collects (inputs) the voice of the installed base from the array microphone 25 and picks up (inputs) the image from the camera 28. The array microphone 25 is formed by arranging two or more omnidirectional microphones side by side. Although details will be described later, the array microphone 25 can set the directivity direction and the sound collection range by electrical control. In the present embodiment, three microphones are used for the array microphone 25.

  As the camera 28, for example, a single focus digital camera equipped with an image sensor such as a CMOS or a CCD is used. The conference terminal 1 according to the present embodiment is used, for example, by being placed on a table, and operations such as panning and tilting for adjusting the imaging direction of the camera 28 are manually performed on the housing 4 of the conference terminal 1. It is done by moving. Moreover, the zoom in the conference terminal 1 is electrically performed by so-called digital zoom. More specifically, since the camera 28 of the present embodiment uses a single-focus digital camera, the angle of view is fixed, and zoom is a pseudo-realization realized by performing trimming and enlargement processing on the captured image. Assume that zoom is used. Hereinafter, a range that can be captured by the camera 28 (a range that falls within an image to be captured) is referred to as an imaging range, but the imaging range is represented by an angle range based on the direction in which the camera 28 faces (imaging direction) The angle of view in the optical zoom (the range of angles that can be picked up as the zoom lens moves and the focal length changes) is treated as synonymous. Therefore, the enlargement / reduction operation with respect to the imaging range performed by the digital zoom may be represented by a change in the angle of view for convenience.

  The operation unit 29 of the conference terminal 1 is provided with a power button, a volume control button, a microphone mute button, and the like. Further, the conference terminal 1 is equipped with a USB interface 21 (see FIG. 2) and can be electrically connected to an external device. In the present embodiment, the conference terminal 1 is connected to, for example, a personal computer (hereinafter abbreviated as “PC”) 9. The PC 9 is a general computer that performs various information processing such as data communication and image display.

  The PC 9 shown in FIG. 1 is a notebook PC and includes a display device 6 and an operation unit 7. However, it goes without saying that a desktop PC without devices such as a display device and an operation unit may be used. . The PC 9 and the conference terminal 1 are electrically connected by a USB cable 2. The connection between the PC 9 and the conference terminal 1 is not limited to the USB cable 2, and various connection methods such as wireless communication such as WiFi (registered trademark), optical communication such as infrared rays, and other IEEE 1394 can be used.

  Audio data collected by the array microphone 25 and image data captured by the camera 28 are transmitted to the PC 9 via the USB cable 2. The conference terminal 1 generates sound from the speaker 27 based on the sound data received from the PC 9.

  The user can execute a remote conference (video conference) using images by using the PC 9 and the conference terminal 1. Specifically, the PC 9 transmits the voice and image data input from the conference terminal 1 to a communication device such as a PC disposed at another site via a network 8 such as the Internet (see FIG. 2). At the same time, the PC 9 receives the voice and image data of the other base from the communication device arranged at the other base. The PC 9 causes the display device 6 to display an image of another site based on the received image data. Further, the PC 9 causes the speaker 27 of the connected conference terminal 1 to generate the sound of another base based on the received sound data. As a result, voices and images of a plurality of bases are shared, and the conference proceeds smoothly even when all users are not at the same base.

  The configurations of the PC 9 and the conference terminal 1 can be changed as appropriate. For example, the voice received from another base is generated by a speaker built in the PC 9, and the speaker 27 of the conference terminal 1 may not be used. Further, a small camera may be connected to a PC including an array microphone, a speaker, and a display device, and a video conference may be performed using the PC as a conference terminal. Of course, the PC may incorporate a camera. Alternatively, the conference terminal 1 may further include a function of transmitting audio and image data, and the PC 9 may be used as a device for generating audio and displaying images received from the conference terminal 1 at another site. Of course, the conference terminal 1 does not necessarily have to be used for video conferencing, and only needs to function as a device that picks up images and collects sound, and the PC 9 is based on image and audio data received from the conference terminal 1. What is necessary is just to display an image and generate sound.

  Next, the electrical configuration of the conference terminal 1 will be described with reference to FIG. The conference terminal 1 includes a CPU 11 that controls the conference terminal 1. A ROM 12, a RAM 13, a flash memory 14, and an input / output interface (I / F) 16 are connected to the CPU 11 via a bus 15.

  The ROM 12 stores a program for operating the conference terminal 1, an initial value, and the like. The RAM 13 temporarily stores various information. The flash memory 14 is a nonvolatile storage device. Connected to the input / output interface 16 are a USB interface (I / F) 21, an audio input processing unit 22, a directivity control unit 26, an audio output processing unit 23, a video input processing unit 24, and an operation unit 29. Yes. The USB interface 21 connects the conference terminal 1 to the PC 9. The voice input processing unit 22 processes voice signals from the array microphone 25 input via the directivity control unit 26 to generate voice data. The directivity control unit 26 performs processing for controlling the directivity direction and the sound collection range of the array microphone 25. The audio output processing unit 23 processes the operation of the speaker 27. The video input processing unit 24 processes image signals from the camera 28 to generate image data.

  Here, the operation principle of processing performed in the directivity control unit 26 for controlling the directivity direction and sound collection range of the sound collected by the array microphone 25 will be briefly described. The sound reaching the individual microphones arranged in an array forms a difference in arrival time depending on from which direction the microphones are arranged. For example, when sound arrives from a direction orthogonal to the direction in which the microphones are arranged (for convenience, “front direction”), the sound reaches each microphone at the same time. For this reason, audio signals are output from the individual microphones from the array microphone 25 and are electrically added together by the audio input processing unit 22, thereby outputting an audio amplified to a magnification corresponding to the number of microphones. Will be obtained. On the other hand, when the sound arrives from a direction oblique to the microphone arrangement direction (for the sake of convenience, “oblique direction” is included, including the side), the sound comes earlier as the microphone is closer to the sound source. A time difference (phase shift) occurs in the sound acquired by each microphone. For this reason, the audio gain when the audio signals from the array microphone 25 are electrically added in the audio input processing unit 22 depends on the arrival angle of the audio to each microphone and the arrangement interval (or arrangement position) of the microphones. It becomes smaller than when reaching from the front. Since the arrangement intervals of the individual microphones are known in advance, the direction of the sound source can be obtained by acquiring the time difference of the sound acquired by each microphone in the directivity control unit 26 and analyzing it by the CPU 11.

  Further, the directivity control unit 26 can delay the sound collected by the individual microphones of the array microphone 25 and output the delayed sound to the sound input processing unit 22. This means that by controlling the delay time with respect to the output of each microphone, the gain can be maximized when sounds arriving from a predetermined diagonal direction are added. In other words, the array microphone 25 can obtain directivity with respect to a desired direction by electrically controlling and delaying the output from each microphone in the directivity control unit 26.

  Thus, the gain of the output of the array microphone 25 that can obtain directivity by delay control is maximized when it arrives from one direction, and decreases when the sound arrives from a direction slightly deviated from that direction. That is, if the delay control is performed so that the sound collected by the individual microphones is uniformly shifted according to the arrangement interval of the microphones, the array microphone 25 can be controlled to have a narrow directivity, and the sound collection range (directivity direction can be changed). The angle range in which sound can be collected when the center is set can be narrowed. Also, if the delay times of the individual microphones are not uniform and a combination of delay times previously obtained by calculation or the like is applied to the individual microphones, the array microphone 25 is controlled to have a wide directivity, and the sound collection range is set. It can also be widened. Furthermore, if the microphones are divided into several groups and the delay control is different for each group, the array microphone 25 can have a plurality of directivity directions. In the present embodiment, based on such an operating principle, the directivity control unit 26 performs a delay process on the sound collected by each microphone in accordance with the calculation by the CPU 11, so that the direction of the array microphone 25 and the collection direction are collected. The sound range is controlled. In the present embodiment, as described above, the sound collection range is an angular range in a direction in which the array microphone 25 can collect sound with the directional direction as the center.

  Further, in the conference terminal 1 of the present embodiment, the control of the directivity direction and the sound collection range of the array microphone 25 is performed so that the sound emitted by the person reflected in the image captured by the camera 28 can be reliably picked up. This is performed according to the analysis result of the image. Specifically, it is determined whether or not the orientation has been changed by image analysis for determining whether or not a human face is included in the image captured by the camera 28 and rotation (panning) of the camera 28 in the horizontal direction. Image analysis is performed. Image analysis that detects the face of a person, for example, extracts parts with facial features such as eyes, nose and mouth from the image, compares the relative position and size with the template, or analyzes geometrically. This is performed by a known method.

  In the present embodiment, even if the relative position of the part having facial features matches the template, if the size does not satisfy a predetermined size, it is detected as a human face. Not. In other words, even if the image analysis includes a portion having the characteristics of a person's face in the image, if the size is smaller than a predetermined size, it is not determined as a person's face. Thereby, for example, even if a person who is far away from the conference terminal 1 is included in the imaging range of the camera 28 and captured in the image, the person is excluded from the target to be detected as a person. .

  The image analysis for detecting the orientation of the camera 28 is performed by, for example, a known method for detecting whether or not there is a displacement of the arrangement position in the image of the feature object reflected in both the latest image and the previously captured image. As described above, the conference terminal 1 is used, for example, by being placed on a table, and panning and tilting of the camera 28 is performed by manually moving the conference terminal 1 by a conference participant or the like. In other words, the conference terminal 1 is not equipped with a driving device for panning and tilting, and panning and tilting by control according to the operation in the PC 9 is not performed. For this reason, the conference terminal 1 does not detect the imaging direction of the camera 28 using the pan and tilt control mechanism. Therefore, the conference terminal 1 detects a change in the orientation of the camera 28 based on the analysis result of the image captured by the camera 28. The feature object is, for example, one that can detect a closed contour line. In the conference terminal 1, if there is a deviation in the arrangement position of the feature object, the magnitude of deviation (the number of dots in the horizontal direction, etc.) in the image is obtained, The number of rotations in the direction is required. Note that these image analysis methods are merely examples, and various known image analysis methods can be applied.

  Next, a specific processing flow for controlling the directivity direction and the sound collection range of the array microphone 25 in the conference terminal 1 will be described with reference to FIGS. A program for executing the processing shown in FIG. 3 is stored in the ROM 12, and is executed by the CPU 11 according to the program.

  The conference terminal 1 is installed, for example, in a conference room so as to be directed toward the conference participants in a posture during use, and is connected to the PC 9. When the power button of the operation unit 29 is turned on by a user (which may be one of the participants), communication with the PC 9 is started and the conference terminal 1 enters a standby state (S11: NO). When the user further operates the PC 9 to receive an imaging start instruction signal from the PC 9 (S11: YES), the CPU 11 starts imaging by the camera 28 and sound collection by the array microphone 25. Further, the CPU 11 starts generating (outputting) the voice of the other base from the speaker 27 based on the voice data received by the PC 9 from the communication device arranged at the other base.

  In the present embodiment, as shown in FIG. 4, it is assumed that the conference is imaged by the conference terminal 1 installed on the front side of the table 52 disposed in the center of the conference room 50. In the conference room 50, three participants 53, 54, 55 are seated around a table 52 on which documents 51 are placed, a white board 56 is prepared on the right front side of the table 52, and a flower 57 is on the right back. It shall be decorated.

  At the start of imaging, the camera 28 is set not to be zoomed, and an object captured in the maximum angle range that can be captured by the camera 28 is reflected in the image captured by the camera 28. The front direction of the conference terminal 1 is directed to the center of the table 52. In the following description, this direction is referred to as an imaging direction A1 for convenience. A signal from the camera 28 that captures the state of the conference room 50 is input to the video input processing unit 24, and data of the image P1 is generated. In the image P1, a person (participants 53, 54, 55) and an object (document 51, table 52, flower 57) included in an imaging range B1 (shown by a thick solid line) that can be captured by the camera 28 are shown. Yes.

  At the start of imaging by the conference terminal 1, the orientation direction C1 of the array microphone 25 is set to the front direction of the camera 28, that is, the same direction as the imaging direction A1 for convenience (that is, the front direction of the conference terminal 1). Further, the CPU 11 further sets an arithmetic expression or table set in advance according to the operation principle described above based on the imaging direction A1 and the imaging range B1 in order to adjust the sound collection range D1 of the array microphone 25 to the initial angle of view of the camera 28. Perform the calculation by. The directivity control unit 26 sets the delay time of each microphone of the array microphone 25 according to the result of the calculation performed by the CPU 11. The audio signals of the conference room 50 collected by the array microphone 25 in which the directivity direction C1 and the sound collection range D1 are controlled are input to the audio input processing unit 22 and added to generate audio data. The image data generated in the video input processing unit 24 and the audio data generated in the audio input processing unit 22 are transmitted to the PC 9 via the USB cable 2 in a streaming format.

  Next, as shown in FIG. 3, the CPU 11 performs image analysis of the image P <b> 1, detects the face of the person (that is, the participants 53 to 55) shown in the image P <b> 1, and counts the number of detected participants. (S12). From the image P1, the faces of the three participants 53 to 55 (parts having human face characteristics) are recognized. The CPU 11 temporarily stores in the RAM 13 (may be the flash memory 14) assuming that the number of participants in the conference is 3 (S12).

  Image capturing by the camera 28 and sound collection by the array microphone 25 are continuously performed, and the generated image data and sound data are streamed to the PC 9. Even if the conference terminal 1 is horizontally rotated (panned) during that time, the video input processing unit 24 generates image data of an image captured in the direction in which the camera 28 is directed. When the CPU 11 receives a zoom instruction signal from the PC 9 by the user's operation on the PC 9, the video input processing unit 24 performs image trimming and enlargement processing according to the instructed zoom magnification to generate image data. To do. In this case, the angle of view corresponding to the zoom magnification is calculated using a predetermined calculation formula or table, and temporarily stored in the RAM 13 (or the flash memory 14) as the current imaging range.

  Image capturing by the camera 28 and sound collection by the array microphone 25 are continuously performed for a predetermined time (S13: NO, S13), and when the predetermined time has passed (S13: YES), S15 to S30. The process is executed. In the processing of S15 to S30, the directivity direction and sound collection range of the array microphone 25 are controlled. Further, when the processes of S15 to S30 are performed, the latest image is stored in the RAM 13 by the camera 28 and used for image analysis by the CPU 11. Note that each time the processes of S15 to S30 are performed, the RAM 13 stores two images, the latest image and the previously captured image, and the previously stored image is overwritten and erased.

  First, the CPU 11 performs image analysis on the latest image newly captured and stored in the RAM 13, and determines whether or not the face of a person shown in the image has been detected (S15). When the conference terminal 1 is not panned or zoomed and the latest image is, for example, the same image as the image P1 of FIG. 4 captured last time, the CPU 11 has three participants 53 and 54. , 55 are recognized, that is, a person is detected (S15: YES). When the number of detected faces is 3 and it is not less than the number of conference participants stored in S12 (S22: NO), the CPU 11 assumes that all participants are within the imaging range of the camera 28. Then, a process of adjusting the sound collection range of the array microphone 25 to the current angle of view of the camera 28 is performed (S23). That is, as shown in FIG. 4, the CPU 11 sets the directivity direction of the array microphone 25 to the same C1 as the imaging direction A1. Similarly to the above, calculation based on the imaging direction A1 and the imaging range B1 is performed so that the sound collection range becomes D1, and an instruction for setting the delay time of each microphone of the array microphone 25 is given to the directivity control unit 26. Send it out.

  As described above, if all the participants 53 to 55 are reflected in the latest image P1, it can be determined that the sound emitted from all the participants 53 to 55 can be collected by collecting sound from the imaging range B1. Therefore, the CPU 11 sets the imaging direction A1 as the directivity direction C1, and obtains and sets a sound collection range D1 having the same size as the imaging range B1 by calculation. Thereby, the sound which participant 53-55 emits can be collected reliably. Thereafter, the process returns to S13.

  Next, in order to project the whiteboard 56, panning is performed on the conference terminal 1. For example, as shown in FIG. 5, when the imaging direction of the camera 28 is directed to A2, the participants 53 to 55 are within the imaging range B1. May disappear. In this case, the participants 53 to 55 are not shown in the latest image P2, and in S15, the CPU 11 cannot detect the face of the person even if the image P2 is analyzed (S15: NO). If the zoom instruction signal has not been received from the PC 9 (S16: NO), the rotation angle is estimated (imaging) by comparing the latest image P2 stored in the RAM 13 with the image P1 previously captured and stored in the RAM 13. Direction detection) is performed (S17).

  As described above, in the image analysis for detecting the imaging direction (the direction of the camera 28), the CPU 11 similarly detects and arranges the feature (for example, the flower 57) reflected in the previous image P1 in the latest image P2. This is performed by a known method for detecting whether there is a shift in position. As shown in FIG. 5, in the image P2, the flower 57 is located at the position near the left end, and in the previous image P1, the position is located near the right end, and the position shift occurs as shown by the arrow E1, so that the conference It is detected that the terminal 1 has been panned. Furthermore, since the imaging range B1 (angle of view) is known, the size of the pan made on the conference terminal 1 from the size of the positional shift of the flower 57 in the images P1 and P2 with respect to the horizontal width of the images P1 and P2. That is, the rotation angle of the conference terminal 1 is calculated.

  When the rotation angle of the conference terminal 1 can be estimated (calculated) (S17: YES), the CPU 11 changes the direction of sound collection in the direction before rotation by the calculated rotation angle from the current imaging direction A2. Return and set the direction to the pointing direction. In the example shown in FIG. 5, since the imaging direction has been found to be directed from A1 (see FIG. 4) to A2 (see FIG. 5) by the above image analysis, the directing direction is set to C1 before rotation. . Further, the CPU 11 sets the sound collection range of the array microphone 25 to D1, which is the sound collection range before rotation (S20).

  Thus, if the participants 53 to 55 shown in the previous image P1 are not shown in the latest image P2, it can be determined that only the conference terminal 1 is panned. Therefore, the CPU 11 matches the directivity direction and the sound collection range of the array microphone 25 with the directivity direction and the sound collection range before the rotation when the rotation angle due to pan can be determined from the image analysis. Thereby, the sound which participant 53-55 emits can be collected reliably. That is, even if the participants 53 to 55 are not shown in the image P3 to show the whiteboard 56, it is possible to reliably collect the sounds emitted by the participants 53 to 55. Thereafter, the process returns to S13.

  Further, when panning for projecting the whiteboard 56 to the conference terminal 1 is performed, for example, as shown in FIG. 6, the imaging direction of the camera 28 is within the imaging range B1, and the participants 53 to 55 are also characteristic features. (Flower 57) may not be included and may be directed to A3. In this case, the CPU 11 cannot detect the face of a person from the latest image P3 (S15: NO). If the zoom instruction signal has not been received (S16: NO), the rotation angle is estimated (detection of the imaging direction) as described above (S17). The CPU 11 cannot detect the feature (flower 57) shown in the previous image P1 (see FIG. 4) in the latest image P3 by the above image analysis, and therefore cannot estimate the rotation angle. Judgment is made (S17: NO).

  In this case, the CPU 11 sets C3, which is the opposite direction of the current imaging direction A3, as the directing direction, and the sound collection range of the array microphone 25 from the entire range of 360 ° to the range of the angle of view of the current camera 28. Except for a certain imaging range B1, the remaining range is set to D3 (S18). In other words, the directivity of the array microphone 25 is set outside the angle of view of the camera 28.

  Thus, if the participants 53 to 55 shown in the previous image P1 are not shown in the latest image P3, it can be determined that only the conference terminal 1 is panned as described above. At this time, if the rotation angle due to pan is not known from the image analysis, the CPU 11 sets the directivity direction and the sound collection range of the array microphone 25 outside the current angle of view of the camera 28. As a result, sound is not collected from a range where it is known that there are no participants 53 to 55, and sound can be collected from other ranges. That is, even if the participants 53 to 55 are not shown in the image P3 to show the whiteboard 56, it is possible to reliably collect the sounds emitted by the participants 53 to 55. Thereafter, the process returns to S13.

  Next, for example, as shown in FIG. 7, the CPU 11 receives a zoom instruction signal from the PC 9, and performs the trimming and enlargement processing of the captured image P <b> 1. 55 may not be included. In this case, the CPU 11 cannot detect a human face from the latest image P4 (S15: NO). Since the zoom instruction signal is received (S16: YES), the process proceeds to S21, and the directivity direction of the array microphone 25 is set to the directivity direction C1 before zooming. Then, the sound collection range of the array microphone 25 is the remaining range except for the imaging range B4 which is the range of the angle of view reduced by the zoom from the imaging range B1 which is the range of the angle of view of the camera 28 before zooming. (S21).

  Thus, if the participants 53 to 55 shown in the previous image P1 are not shown in the latest image P4, and the CPU 11 receives a zoom signal at that time, the angle of view is reduced by zooming. Therefore, it can be determined that the participants 53 to 55 are no longer shown in the image P4. Therefore, the CPU 11 sets the sound collection range of the array microphone 25 to the sound collection range D4 that is a range obtained by removing the current image pickup range B4 that is known to have no participants 53 to 55 from the image pickup range B1 before zooming. Thereby, even when the participants 53 to 55 no longer appear in the image P4 after zooming, it is possible to reliably collect the sounds emitted by the participants 53 to 55. Thereafter, the process returns to S13.

  Next, processing when panning or zooming is performed in the conference terminal 1 and the number of participants appearing in the captured image is reduced will be described. Even if the CPU 11 detects a human face in the image in S15 (S15: YES), if the number is smaller than the number of conference participants stored in S12 (S22: YES), S25 to S30. Processing is performed.

  For example, as illustrated in FIG. 8, the CPU 11 that has received the zoom instruction signal performs trimming and enlargement processing of the image P <b> 1, and as a result, some participants 53 and 54 are included in the imaging range B <b> 5 that is reduced by zooming. May be. In this case, the CPU 11 can detect a person's face from the latest image P5 (S15: YES), the number of persons is smaller than the number of participants (S22: YES), and a zoom instruction signal has been received (S25: YES). ), Go to S30. The CPU 11 sets the directivity direction of the array microphone 25 to the directivity direction C1 before zooming, and the sound collection range of the array microphone 25 is the same as the imaging range B1 that is the range of the angle of view of the camera 28 before zooming. Set to D1 (S30).

  In this way, if some of the participants 53 and 54 appear in the latest image P5 and the CPU 11 receives a zoom signal at that time, the participant 55 has the angle of view narrowed by zooming. It can be determined that the image is no longer displayed in the image P5. Therefore, the CPU 11 sets the sound collection range of the array microphone 25 to the same sound collection range D4 as the imaging range B1 before zooming. As a result, it is possible to reliably collect the sounds produced by the participants 53 and 54 shown in the zoomed image P5 and the participants 55 who are not shown. Thereafter, the process returns to S13.

  By the way, the conference terminal 1 is panned, and as a result, only some participants may be included in the imaging range. For example, as illustrated in FIG. 9, when the imaging direction of the camera 28 is directed to A6 where the participant 54 is included in the imaging range B1 and the characteristic object (flower 57) is included, the CPU 11 performs image analysis. Thus, the face of the person (participant 54) is detected from the image P6 (S15: YES). Since the faces of the other participants 53 and 55 that are not shown in the image P6 cannot be detected, the number of persons to be detected is smaller than the number of participants stored in S12 (S22: YES).

  If the zoom instruction signal is not received (S25: NO), the CPU 11 estimates the rotation angle (detects the imaging direction) (S26). The feature (flower 57) shown at the position near the right end of the previous image P1 (see FIG. 4) is shown at a position slightly to the left of the center in the image P6, and there is a displacement as shown by the arrow E2. From this, it is detected that the conference terminal 1 has been panned. Further, based on the imaging range B1, the size of the pan made to the conference terminal 1, that is, the rotation angle of the conference terminal 1 is calculated from the size of the positional shift of the flower 57 in the images P1 and P6.

  When the rotation angle of the conference terminal 1 can be estimated (calculated) (S26: YES), the CPU 11 sets the directivity direction of the array microphone 25 in an intermediate direction between the image pickup direction A6 and the directivity direction C1 before the rotation. Set to a certain C6. Then, the sound collection range of the array microphone 25 is set to D6 obtained by adding the field angle range of the current imaging range B1 of the camera 28 with respect to the imaging direction A6 and the sound collection range D1 with respect to the previous directing direction C1 ( S28).

  As described above, when a part of the participants 54 is reflected in the latest image P6 by panning, it is possible to determine that the conference terminal 1 has been panned because the participants 54 are reflected. Therefore, when the rotation angle by pan is known from the image analysis, the CPU 11 sets the directivity direction of the array microphone 25 to an intermediate direction between the directivity direction before the rotation and the imaging direction after the rotation of the camera 28. Then, the sound collection range of the array microphone 25 is adjusted to the range obtained by adding the image pickup range based on the image pickup direction after the rotation to the sound collection range before the rotation of the camera 28. As a result, it is possible to reliably collect the voices uttered by the participant 54 noted by panning and the participants 53 and 55 that are no longer shown in the image P6. Thereafter, the process returns to S13.

  Further, when one participant 54 explains using the whiteboard 56, the conference terminal 1 may be panned so as to reflect the participant 54 in accordance with the movement from the original position. For example, as shown in FIG. 10, the imaging direction of the camera 28 is directed to A7 where the participant 54 is included in the imaging range B1 but the feature (flower 57) is not included. Similarly to the above, the CPU 11 detects the face of the person (participant 54) from the image P7 by image analysis (S15: YES), but the number of persons to be detected is smaller than the number of participants (S22: YES).

  The CPU 11 determines that the rotation angle cannot be estimated unless the feature (flower 57) shown in the previous image P1 (see FIG. 4) can be detected in the latest image P7 (see FIG. 4). S26: NO). In this case, the CPU 11 sets the directivity direction of the array microphone 25 to 360 ° in all directions (non-directional), and sets the sound collection range of the array microphone 25 to D7 which is the full range of 360 °.

  As described above, when the rotation angle due to pan is not known from the image analysis, the CPU 11 sets the sound collection range D7 and collects sound from the entire 360 ° range, so that the sound emitted from the participant 54 shown in the image P7 is generated. In addition to this, it is possible to deal with voices uttered by the participants 53 and 55 not shown in the image P7. In other words, it is possible to reliably collect the voices uttered by the participant 54 noted by panning and the participants 53 and 55 no longer appearing in the image P7. Thereafter, the process returns to S13.

  As described above, in the conference terminal 1 according to the present embodiment, if a person is not included in the imaging range of the conference terminal 1, at least a part of the area outside the imaging range may be set as a sound collection target area. Therefore, the area where the person is present can be included in the sound collection range, and the sound emitted by the person can be reliably collected. In addition, since the area within the imaging range that does not include a person is excluded from the sound collection target area, the sound is not collected even if there is noise or the like having a generation source in that area. Sound can be collected more clearly.

  When the imaging range changes, the person is expected to be in the imaging range before the change. Therefore, if the direction of the array microphone 25 and the sound collection range are controlled based on the contents of the change in the imaging range, the area where the person is present can be surely included in the sound collection target area. Therefore, it is possible to reliably and more clearly collect the voice uttered by the person.

  When the imaging range changes, the person is expected to be in the imaging range before the change, but if the imaging direction after the change cannot be specified from before the change, based on the imaging direction after the change, The directivity direction and sound collection range before the change cannot be specified. Therefore, by controlling the directivity direction and the sound collection range of the array microphone 25 and collecting sounds from all areas outside the imaging range of the conference terminal 1 among the areas that can be collected, the area where the person is present can be obtained. It is possible to collect sound from a person surely and more clearly, because it can be surely included in the sound collection target area, but not collected from an area known to have no person. it can.

  When the imaging range changes, the person is expected to be in the imaging range before the change, and if the imaging direction after the change can be specified from before the change, the person changes based on the imaging direction after the change. The previous directivity direction and sound collection range can be specified. Therefore, by controlling the directivity direction and sound collection range of the array microphone 25 and collecting sound from the sound collection target region of the array microphone 25 before the change of the imaging range, it is possible to reliably collect the region where the person is present. It is possible to collect sound from a person reliably and more clearly while avoiding collecting from an area where there is no person while making it a target area.

  When the imaging range changes, if the change is caused by a change in the angle of view, the person is expected to be in an area excluding the imaging range after the change from the imaging range before the change. Therefore, the directivity direction and the sound collection range of the array microphone 25 are controlled, and the area that is the sound collection target of the array microphone 25 before the change of the angle of view is excluded from the area excluding the area that overlaps the imaging range after the change of the angle of view. By collecting sound, it is possible to reliably collect sound from a person while avoiding collecting from an area without a person, while ensuring that the area with the person is the area to be collected. it can.

  If a part having a human face characteristic within a captured image that is less than or equal to a predetermined size is not detected as a person, a person who is not an imaging target by chance is included in the imaging range. Even if it is, the person does not become a control condition of the array microphone 25. As a result, it is possible to prevent an incorrect directivity direction and sound collection range from being set, and it is possible to reliably collect sound produced by the person to be collected.

  The present invention is not limited to the above embodiment, and various modifications can be made. A single-focus digital camera was used as the camera 28, and zooming was performed by a pseudo digital zoom realized by performing trimming and enlargement processing on the captured image. The camera 28 mechanically changes the focal length. A zoom lens may be provided to realize optical zoom.

  The array microphone 25 is provided with three microphones as an example. However, the number may be two or more, preferably three or more. The larger the number, the more accurately the sound collection range is set. be able to. Moreover, although the omnidirectional microphone was used in the present embodiment for each microphone constituting the array microphone 25, a directional microphone may be used. Alternatively, the array microphone 25 may be configured by combining an omnidirectional microphone and a directional microphone.

  The calculation of the rotation angle of the pan of the conference terminal 1 was performed by detecting the position of the feature from the images before and after the rotation by image analysis. The conference terminal 1 is provided with an acceleration sensor, and the orientation of the conference terminal 1 It may be possible to keep track of at any time. In addition, as a feature, markers may be provided at several places in the conference room 50 so that the orientation of the conference terminal 1 can be grasped from the markers reflected in the image by image analysis. Considering the cost of providing an acceleration sensor and the effort of preparing a marker, it is preferable to adopt a method of grasping the orientation of the conference terminal 1 by image analysis as in the present embodiment, because it can be processed only by software.

  The installation direction of the conference terminal 1 may be any direction. For example, the conference terminal 1 may be tilted 90 degrees and attached to a wall or the like, and the pan in the present embodiment may correspond to a tilting operation. In this case, if the movement of the feature in the image in the vertical direction is detected by image analysis and the rotation angle is obtained, the directivity direction and the sound collection range of the array microphone 25 can be controlled.

  In the present embodiment, the conference terminal 1 corresponds to the “imaging device” of the present invention. The camera 28 corresponds to “imaging means”. The array microphone 25 corresponds to “sound collecting means”. The CPU 11 that performs calculations for determining the directivity direction and the sound collection range of the array microphone 25 according to various conditions, and the array microphone 25 by controlling the delay time of each microphone of the array microphone 25 based on the calculation results of the CPU 11. The directivity control unit 26 that controls the directivity direction and the sound collection range corresponds to “control means”. The CPU 11 that detects a person in S15 and determines whether or not a person is included in the image corresponds to the “first determination unit”. The CPU 11 that determines that a person cannot be detected in S15, and that has detected a person but determines that the number of persons has decreased in S22 corresponds to “second determination means”.

1 Conference terminal 11 CPU
13 RAM
25 Array microphone 26 Directivity control unit 28 Camera

Claims (8)

An imaging means for capturing an image;
A plurality of sound collecting means configured to be integrated with the imaging means and collecting sound;
Control means for controlling a directivity direction and a sound collection range for collecting sound by the plurality of sound collection means based on an image pickup range of the image by the image pickup means;
First determination means for determining whether or not a person is included in the imaging range based on an image captured by the imaging means;
With
When the first determination unit determines that no person is included in the imaging range, the control unit includes a region outside the imaging range of the imaging unit in a region where the sound collection unit can collect sound. An imaging apparatus that controls a directivity direction and a sound collection range of the sound collecting means so that at least a part of the area is a sound collection target area of the sound collecting means. Further comprising second determination means for determining whether or not the imaging range of the imaging means has changed,
The control means controls the directivity direction and the sound collection range of the sound collection means based on the content of the change of the image pickup range when the second determination means determines that the image pickup range has changed. The imaging apparatus according to claim 1.   The control means is determined by the second determination means that the imaging range has changed, and the change in the imaging range is caused by a change in the imaging direction of the imaging means, When it is determined that the imaging direction after the change from the imaging direction cannot be specified, all the areas outside the imaging range of the imaging means among the areas where the sound collecting means can collect sound are The imaging apparatus according to claim 2, wherein a directivity direction and a sound collection range of the sound collection unit are controlled so as to be a region where sound is collected by the sound collection unit.   The control means is determined by the second determination means that the imaging range has changed, and the change in the imaging range is caused by the change in the imaging direction of the imaging means, When it is determined that the imaging direction after the change from the imaging direction can be specified, the sound is collected from the sound collection target area of the sound collecting means before the change of the imaging range. The imaging apparatus according to claim 2 or 3, wherein a directivity direction and a sound collection range of the sound collecting means are controlled.   When the second determination unit determines that the imaging range has changed and the control unit determines that the change in the imaging range is caused by a change in the angle of view of the imaging unit, the control unit A region obtained by excluding a region that overlaps the imaging range after the change of the angle of view from a region to be collected by the sound collecting unit before the change of the sound field becomes a region to be collected by the sound collecting unit. The imaging apparatus according to claim 2, wherein a directivity direction and a sound collection range of the sound collecting means are controlled.   The first determination unit recognizes a part having a human facial feature from an image captured by the imaging unit, and a person is included in the imaging range when the size of the recognized part is larger than a predetermined size. The imaging apparatus according to claim 1, wherein the imaging apparatus is determined to be capable of being detected. An imaging method that is executed in a computer in order to cause an imaging device in which an imaging unit that captures an image and a plurality of sound collection units that collect sound to function together,
A control step for controlling a directivity direction and a sound collection range for collecting sound by the plurality of sound collection means based on an image pickup range of the image by the image pickup means;
A first determination step of determining whether or not a person is included in the imaging range based on an image captured by the imaging unit;
Including
Further, when it is determined in the first determination step that no person is included in the imaging range, in the control step, the imaging range of the imaging unit among the areas where the sound collecting unit can collect sound. An imaging method, wherein a directivity direction and a sound collection range of the sound collecting means are controlled so that at least a part of an outside area is a sound collection target area of the sound collecting means. A program for functioning an imaging apparatus in which an imaging unit that captures an image and a plurality of sound collection units that collect sound are configured,
On the computer,
A control step for controlling a directivity direction and a sound collection range for collecting sound by the plurality of sound collection means based on an image pickup range of the image by the image pickup means;
A first determination step of determining whether or not a person is included in the imaging range based on an image captured by the imaging unit;
And execute
Further, when it is determined in the first determination step that no person is included in the imaging range, in the control step, the imaging range of the imaging unit among the areas where the sound collecting unit can collect sound. A program in which a directivity direction and a sound collection range of the sound collecting means are controlled so that at least a part of an outside area is a sound collection target area of the sound collecting means.

Images