JP2012186551A - Control device, control system, and control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the quality of sound collection.SOLUTION: A control device for controlling image acquisition means capable of a horizontal panning operation and sound collection means with a changeable sound collection range comprises sound collection range change means for changing the sound collection range of the sound collection means using a distance between the position of the image acquisition means and the position of a subject captured in an image acquired by the image acquisition means, a panning direction of the image acquisition means, a distance between the position of the image acquisition means and the position of the sound collection means, and a direction of the position of the sound collection means with reference to the position of the image acquisition means.

Description

  The present invention relates to a technique for controlling a sound collecting device.

  2. Description of the Related Art Conventionally, there is a technology of a video conference apparatus in which a microphone directivity characteristic, a camera shake angle, and a zoom angle are linked by a built-in video camera or the like (Patent Document 1).

JP-A-10-155107

  However, in the above technology, since the camera and the microphone are integrated, a subject (participant) that can be photographed when the microphone is installed at the center of the conference table or the like to improve the sound collection quality in a video conference or the like. It will be limited.

  An object of the present invention is to provide a technique for improving the quality of sound collection.

  In order to solve the above problems, a control device according to the present invention is a control device that controls image acquisition means capable of panning left and right and sound collection means capable of changing a sound collection range, wherein the image The distance between the position of the acquisition means and the position of the subject reflected in the image acquired by the image acquisition means, the pan direction of the image acquisition means, the position of the image acquisition means, and the position of the sound collection means A sound collection range changing means for changing a sound collection range of the sound collection means using a distance between the sound acquisition means and a direction of the position of the sound collection means with respect to the position of the image acquisition means. It is characterized by.

  In addition, the control system according to the present invention includes an imaging device including a left and right pan operation unit, a sound collection device including a sound collection range changing unit, a control device that controls the imaging device and the sound collection device, The control device includes a distance between a position of the imaging device and a position of a subject reflected in an image acquired by the imaging device, a pan direction of the imaging device, and a position of the imaging device. A collection that instructs to change the sound collection range of the sound collection device using a distance between the sound collection device position and a direction of the sound collection device position with respect to the position of the imaging device. A sound range change instruction means is provided.

  Further, an imaging apparatus according to the present invention includes pan operation means for performing a pan operation to the left and right, an angle of view change operation means for performing an operation of changing an angle of view being photographed, and pan direction information for specifying a pan direction. Pan direction information acquisition means for acquiring, distance information acquisition means for acquiring distance information for specifying the distance to the subject, angle of view information acquisition means for acquiring angle of view information for specifying the angle of view being shot, Output means for outputting pan direction information, the distance information, and the angle of view information, and the output means finishes any of the operations of the pan operation means and the angle of view change operation means. The output is performed when a predetermined time has elapsed.

  The control method according to the present invention is a control method for a control device that controls image acquisition means capable of panning left and right and sound collection means capable of changing a sound collection range, wherein the control device includes: The distance between the position of the image acquisition means and the position of the subject reflected in the image acquired by the image acquisition means, the pan direction of the image acquisition means, the position of the image acquisition means, and the sound collection A sound collection range changing step of changing the sound collection range of the sound collecting means using a distance between the sound collecting means and a direction of the sound collecting means relative to the position of the image acquiring means. It is characterized by carrying out.

  According to the present invention, the quality of sound collection can be improved while the camera and the microphone can be arranged independently.

It is a figure which shows the structural example of the video conference system which concerns on embodiment of this invention. It is a figure which shows the function structural example of a video conference system. It is a figure which shows the hardware constitutions of a video conference system. It is a figure which shows the structure of a microphone array apparatus. It is a figure which shows the example of arrangement | positioning of a video conference system. It is a figure which shows the structural example of a microphone array position table. It is a figure which shows the structural example of a microphone array output level table. It is a sequence diagram of a microphone array setting process. It is explanatory drawing explaining the example of a screen of the initialization process of microphone array position setting. It is explanatory drawing explaining the mechanism of microphone array setting calculation. It is a figure which shows the structural example of an output level adjustment table. It is explanatory drawing explaining the mechanism of microphone array output level adjustment.

  Hereinafter, a video conference system 1 including a microphone control device to which a first embodiment of the present invention is applied will be described with reference to the drawings.

  FIG. 1 is a diagram showing a video conference system 1 according to an embodiment of the present invention. The video conference system 1 according to the present invention includes a server 80 and a conference terminal A20, a conference terminal B21, and a conference terminal C22 that can be connected to the server 80 via the network 70. Basically, it is assumed that the conference terminal A20 is arranged in the conference room A10, the conference terminal B21 is arranged in the conference room B11, and the conference terminal C22 is arranged in the conference room C12.

  A camera A30, a microphone array A40, a display A50, and a speaker A60 that can communicate with the conference terminal A20 are connected to the conference terminal A20. The conference terminal A20 can operate as a microphone control device.

  The camera A30 is a so-called image acquisition device, such as a video camera, that acquires video information as digital data and transmits it to the conference terminal A20, receives an instruction of a shooting range from the conference terminal A20, and performs shooting.

  The microphone array A40 collects ambient sounds, acquires them as digital data, transmits them to the conference terminal A20, receives a command from the conference terminal A20, such as the sound collection direction and the sound collection sensitivity, and collects the sound. It is. As will be described later, the microphone array A40 is a device in which a plurality of microphones that collect a predetermined range are arranged so as to have a sound collection range that extends radially from the installation position of the microphone array A40, and gain adjustment is performed for each microphone. Is possible.

  The display A50 displays and outputs display information transmitted from the connected conference terminal A20. The display information includes video information acquired by the conference terminal A20 via the camera A30, video information received via the network 70 from another conference terminal B21 or conference terminal C22, and the like. In addition, the display A50 can also display a setting screen such as a setting value transmitted from the conference terminal A20.

  The speaker A60 outputs audio information transmitted from the connected conference terminal A20. The audio information includes audio information acquired by the conference terminal A20 via the microphone array A40, audio information received via the network 70 from another conference terminal B21 or conference terminal C22, and the like. In addition, the speaker A60 can output voice such as guidance transmitted from the conference terminal A20.

  A camera B31, a microphone array B41, a display B51, and a speaker B61 that can communicate with the conference terminal B21 are connected to the conference terminal B21. The conference terminal B21 can operate as a microphone control device.

  The camera B31 is a so-called image acquisition device, such as a video camera, that acquires video information as digital data and transmits it to the conference terminal B21, receives an instruction of a shooting range from the conference terminal B21, and performs shooting.

  The microphone array B41 collects ambient sounds, acquires them as digital data, transmits them to the conference terminal B21, and receives a command from the conference terminal B21 such as the sound collection direction and the sound collection sensitivity to collect sound. It is. As will be described later, the microphone array B41 is a device in which a plurality of microphones that collect sound within a predetermined range are arranged so as to have a sound collection range radially extending from the installation position of the microphone array B41, and gain adjustment is performed for each microphone. Is possible.

  The display B51 displays and outputs display information transmitted from the connected conference terminal B21. The display information includes video information acquired by the conference terminal B21 via the camera B31 or video information received via the network 70 from another conference terminal A20 or conference terminal C22. In addition, the display B51 can also display a setting screen such as a setting value transmitted from the conference terminal B21.

  The speaker B61 outputs audio information transmitted from the connected conference terminal B21. The audio information includes audio information acquired by the conference terminal B21 via the microphone array B41, audio information received via the network 70 from another conference terminal A20 or the conference terminal C22, and the like. In addition, the speaker B61 can output voice such as guidance transmitted from the conference terminal B21.

  A camera C32, a microphone array C42, a display C52, and a speaker C62 that can communicate with the conference terminal C22 are connected to the conference terminal C22. The conference terminal C22 can operate as a microphone control device.

  The camera C32 is a so-called image acquisition device, such as a video camera, that acquires video information as digital data and transmits it to the conference terminal C22, receives an instruction of a shooting range from the conference terminal C22, and performs shooting.

  The microphone array C42 collects ambient sound, acquires it as digital data, transmits it to the conference terminal C22, receives a command from the conference terminal C22, such as the sound collection direction and the sound collection sensitivity, and collects the so-called microphone. It is. As will be described later, the microphone array C42 is a device in which a plurality of microphones that collect a predetermined range are arranged so as to have a sound collection range radially extending from the installation position of the microphone array C42, and gain adjustment is performed for each microphone. Is possible.

  The display C52 displays and outputs display information transmitted from the connected conference terminal C22. The display information includes video information acquired by the conference terminal C22 via the camera C32 or video information received via the network 70 from another conference terminal A20 or conference terminal B21. In addition, the display C52 can also display a setting screen such as a setting value transmitted from the conference terminal C22.

  The speaker C62 outputs audio information transmitted from the connected conference terminal C22. The audio information includes audio information acquired by the conference terminal C22 via the microphone array C42, audio information received via the network 70 from another conference terminal A20 or conference terminal B21, and the like. In addition, the speaker C62 can output voice such as guidance transmitted from the conference terminal C22.

  In the present embodiment, the network 70 is assumed to be a wide area network based on TCP / IP or the like such as the Internet, but is not limited thereto, and is not limited to a LAN (Local Area Network) or a WAN (Wide Area Network). It may be a network.

  The server 80 implements a video conference service between the conference terminal A20, the conference terminal B21, and the conference terminal C22 via the network 70. Specifically, for example, the server 80 configures and manages a virtual conference room between conference terminals targeted for participation in the conference, and mediates communication between the conference terminals targeted for participation. In the present embodiment, the server 80 does not acquire the video data and audio data of the conference terminal and distribute them to other conference terminals, but constructs a communication path for the video data and audio data between the conference terminals. Stay on. However, the present invention is not limited to this, and video data and audio data between conference terminals may be intensively acquired and distributed to other conference terminals participating in the conference.

  FIG. 2 is a diagram illustrating a functional configuration example of the camera A30, the conference terminal A20, and the microphone array A40 arranged in the conference room A10 of the video conference system 1. The camera B31, the conference terminal B21, the microphone array B41, the camera C32, the conference terminal C22, and the microphone array C42 for the conference room B11 and the conference room C12 basically have the same functional configuration. In addition, the display A50, the speaker A60, the display B51, the speaker B61, the display C52, and the speaker C62 are not illustrated, but have normal functional configurations.

  As shown in FIG. 2, the camera A30 includes a pan / zoom control unit 310, a camera information notification unit 320, and a camera operation enable / disable control unit 330. The pan / zoom control unit 310 controls the angle of view and direction in which the camera A30 shoots, that is, the range that appears in the video. Specifically, when the pan / zoom control unit 310 receives an instruction to change the deflection angle to the left and right, the pan / zoom control unit 310 changes the deflection angle of the optical device 35 to be described later according to the instruction. When an instruction to change the angle of view is received, the angle of view of the optical device 35 to be described later is changed according to the instruction.

  Note that the pan / zoom control unit 310 cannot follow the pan or zoom change instruction, for example, if the shake angle change instruction exceeds the physical movable range of the optical device 35, or, for example, According to the change instruction of the angle of view, when the optical or image processing zoom (optical zoom, digital zoom, or the like) of the optical device 35 is exceeded, control is performed so as to follow the instruction within a changeable range.

  The camera information notification unit 320 acquires information about the swing angle (pan direction), the distance to the subject, and the angle of view at a predetermined timing, and outputs the information to the connected conference terminal A20. In the present embodiment, the camera information notification unit 320 determines the shake angle (pan direction), subject when the pan / zoom operation is not performed for a predetermined time (for example, 3 seconds) after the pan / zoom operation is performed. It is assumed that information about the distance and angle of view is acquired and output to the connected conference terminal A20. In this embodiment, the deflection angle (pan direction) is obtained from the difference between the direction of the pedestal (front) and the direction of the optical axis of the camera, and the distance to the subject is focused on the subject at a predetermined distance measuring point. Is a distance obtained from the focal length in a state where the two are matched, and the angle of view is an angle of view obtained according to the amount of extension of the lens group.

  The camera operation valid / invalid control unit 330 controls validity / invalidity of accepting an operation from the conference terminal A20 for the camera A30. Specifically, when the camera operation enable / disable control unit 330 receives a camera operation lock instruction from the conference terminal A20, the camera operation enable / disable control unit 330 performs operation lock so that a pan / zoom change instruction to the pan / zoom control unit 310 is not received. When the camera operation enable / disable control unit 330 receives a camera operation lock release instruction from the conference terminal A20, the camera operation enable / disable control unit 330 releases the operation lock so as to receive a pan / zoom change instruction to the pan / zoom control unit 310.

  Although not shown, the camera A30 is an image acquisition control unit that receives an instruction to start image acquisition and acquires an image, and receives an instruction to end image acquisition and ends image acquisition in the same manner as a normal camera. An acquisition end control unit, an image output control unit that outputs the acquired image to an external device such as the conference terminal A20 or a storage medium such as a memory card attached to the camera A30, a distance measurement unit that specifies the distance to the subject, and an optical device A control unit such as a focus control unit for adjusting the focal position (focus) of 35 is provided. The camera A30 can perform panning and zooming by receiving panning and zooming operations from an operator (or conference participant) of the conference room A10 or an operator of the conference partner (or conference participant).

  The conference terminal A20 includes a camera state detection unit 210, a microphone array control unit 220, a microphone array position data memory 230, a camera operation enable / disable instruction unit 240, and a camera operation instruction unit 250. The camera state detection unit 210 detects the state of the connected camera A30. Specifically, the camera state detection unit 210 acquires information about the shake angle (pan direction), the distance to the subject, and the angle of view from the connected camera A30. Furthermore, when the microphone array A40 in the image photographed by the camera A30 is at a position within a predetermined screen, the camera state detection unit 210 determines the microphone array A40 according to the swing angle of the camera A30 and the distance to the subject. Specify the position of.

  The microphone array control unit 220 performs gain adjustment for each microphone of the microphone array A40 according to the position of the microphone array A40, the position of the subject of the camera A30, and the angle of view. Specifically, the microphone array control unit 220 stores microphone array position data in which the relative position of the microphone array A40 (the pan angle from the front direction of the camera A30 and the linear distance from the camera A30 to the microphone array A40) is stored. The microphone position is acquired from the memory 230.

  Then, the microphone array control unit 220 uses the direction (identified based on the shake angle) of the subject imaged by the camera A30, the distance to the subject, the angle of view, and the microphone position to determine the microphone position. Identify the direction of the subject as seen from the perspective. The microphone array control unit 220 outputs a microphone gain adjustment instruction including a predetermined microphone gain adjustment value to the microphone array A40 for each microphone of the microphone array A40 according to the direction of the subject viewed from the microphone position. In the present embodiment, the output level of the microphone amplifier is not adjusted when the camera angle of view is 60 ° or more. However, when the camera angle of view is less than 60 °, the microphone output level is increased or decreased to control the sound collection range.

  The microphone array position data memory 230 stores information for specifying the position of the microphone array A40. Specifically, the microphone array position data memory 230 includes a microphone array position table 600 shown in FIG. 6 and a microphone array output level table 650 shown in FIG.

  The microphone array position table 600 stores the position of the microphone array. In the microphone array position table 600, information for specifying the distance between the camera A and the microphone array A is stored. The distance 601 between the camera A and the microphone array A and the panning direction of the microphone array as viewed from the front of the camera are specified. A reference pan direction angle 602. For example, when the reference pan direction angle 602 is “−5 °”, it indicates that the microphone array exists in the minus 5 ° direction from the front of the camera A30, that is, in the direction panned 5 ° to the left.

  The microphone array output level table 650 stores information for adjusting the output level for each microphone constituting the microphone array. The microphone array output level table 650 includes a camera angle of view 651 that is an angle of view captured by the camera A30 and an output level of the microphone selected as the sound collection direction set as the output level of the microphone selected as the sound collection direction. 652, the output level of the microphone adjacent to the selected microphone set as the output level of the microphone adjacent to the selected microphone, and the output level of the other microphones excluding the selected microphone and the adjacent microphone Other microphone output levels 654 are included. The output level stored in the microphone array output level table 650 corresponds to the camera angle of view when the output level when the microphone array A40 is not controlling the sound collection range (360 ° sound collection state) is 0 dB. This specifies the output level of each microphone amplifier.

  The camera operation enable / disable instruction unit 240 instructs an operation from the conference terminal A20 to the camera A30. Specifically, the camera operation enable / disable instruction unit 240 instructs the camera A30 to perform a camera operation lock instruction / camera operation lock release.

  The camera operation instruction unit 250 instructs an operation from the conference terminal A20 to the camera A30 or the other camera B31 or camera C32 in the conference via the network 70. Specifically, when the camera operation instruction unit 250 receives a pan / zoom operation of a designated camera among the cameras connected to the conference terminals participating in the same conference, the camera operation instruction unit 250 Outputs pan / zoom change instructions. At this time, if a camera connected to another conference terminal is an operation target, information including change instruction information is transmitted to the network 70 with the conference terminal as a destination.

  When the camera operation instruction unit 250 receives a camera pan / zoom change instruction via the network 70, the camera operation instruction unit 250 outputs a pan / zoom change instruction corresponding to the change instruction to the connected camera A30. To do. By doing in this way, it is possible to perform pan / zoom of the camera in the destination conference room or the own conference room via each conference terminal participating in the conference.

  The microphone array A40 is obtained from the microphone gain control unit 410, the microphones 420a to 420l, the microphone amplifiers 430a to 430l that adjust the gains of the microphones corresponding to the microphones 420a to 420l, and the microphone amplifiers 430a to 430l. A mixer 440 that combines the gains, and a speech encoding unit 450 that converts the combined output obtained by the mixer 440 into speech information.

  The microphone gain control unit 410 adjusts the gain of the microphone amplifiers 430a to 430l for each of the microphones 420a to 420l. In the present embodiment, the microphone gain control unit 410 receives the gain adjustment instruction of the microphones 420a to 420l from the conference terminal A20, and adjusts the gain of the corresponding microphone amplifiers 430a to 430l.

  The audio encoding unit 450 samples the audio information output from the mixer 440 at a predetermined frame rate, and outputs audio data. This process is a normal encoding process.

  FIG. 3 is a diagram illustrating a hardware configuration example of the camera A30, the conference terminal A20, and the microphone array A40. The camera B31, the conference terminal B21, the microphone array B41, the camera C32, the conference terminal C22, and the microphone array C42 are assumed to have the same hardware configuration.

  In this embodiment, the camera A30 is a video camera device that operates independently. However, the computer is not limited to this, and may be a computer classified into a PC (personal computer), a workstation, various mobile phone terminals, a PDA (Personal Digital Assistant), a smartphone, and the like.

  The camera A30 includes a bus 131, a calculation device 132, a main storage device 133, a communication device 134, and an optical device 135. The arithmetic device 132 is an arithmetic device such as a CPU (Central Processing Unit).

  The main storage device 133 is a memory device such as a RAM (Random Access Memory).

  The communication device 134 is a communication device that communicates with a device connected to the network 70 such as the Internet or a LAN and a conference terminal A20 connected with a dedicated cable or the like.

  The optical device 135 is a so-called imaging device including, for example, a video camera lens group and a focus driving unit thereof, an imager using a CCD or a CMOS, an image processing unit that performs image generation processing, and the like.

  The arithmetic device 132, the main storage device 133, the communication device 134, and the optical device 135 are connected to each other via a bus 131.

  The pan / zoom control section 310 of the camera A30 described above is realized by the optical device 135. The camera information notification unit 320 and the camera operation enable / disable control unit 330 are realized by a program that causes the arithmetic device 132 to perform processing. This program is stored in the main storage device 133 or a ROM device (not shown), loaded onto the main storage device 133 for execution, and executed by the arithmetic device 132.

  In the present embodiment, the conference terminal A20 is a so-called terminal device that operates independently. However, the computer is not limited to this, and may be a computer classified into a PC (personal computer), a workstation, a server device, various mobile phone terminals, a PDA (Personal Digital Assistant), a smartphone, and the like.

  The conference terminal A20 includes a bus 121, a calculation device 122, a main storage device 123, and a communication device 124. The arithmetic device 122 is an arithmetic device such as a CPU (Central Processing Unit).

  The main storage device 123 is a memory device such as a RAM (Random Access Memory).

  The communication device 124 is a device connected to a network 70 such as the Internet or a LAN, another conference terminal connected by a dedicated cable or the like, and a device connected to the camera A30, microphone array A40, display A50, speaker A60, and the like. A communication device that performs communication.

  The arithmetic device 122, the main storage device 123, and the communication device 124 are connected to each other via a bus 121.

  The camera state detection unit 210, the microphone array control unit 220, and the camera operation enable / disable instruction unit 240 of the conference terminal A20 are realized by a program that causes the arithmetic device 122 to perform processing. This program is stored in the main storage device 123 or a ROM device (not shown), loaded onto the main storage device 123 for execution, and executed by the arithmetic device 122.

  Further, the microphone array position data memory 230 of the conference terminal A20 is realized by being stored in the main storage device 123.

  In the present embodiment, the microphone array A40 is a microphone device that operates independently. The microphone array A40 includes a bus 141, a calculation device 142, a main storage device 143, a communication device 144, and a sound collection device 145. The arithmetic device 142 is an arithmetic device such as a CPU (Central Processing Unit).

  The main storage device 143 is a memory device such as a RAM (Random Access Memory).

  The communication device 144 is a communication device that communicates with a device connected to the network 70 such as the Internet or a LAN and a conference terminal A20 connected with a dedicated cable or the like.

  The sound collector 145 has a structure as shown in FIG. 4, for example. FIG. 4 is a diagram showing the configuration of the microphone array apparatus. The microphone array A40 includes a plurality of microphones 420a to 420l that are equally arranged in a ring shape. In one microphone (microphone 420a in FIG. 4), a reference position mark 460 serving as a reference for the sound collection direction of the microphone array A40 is provided so as to be clearly recognizable in appearance.

  Each of the microphones 420a to 420l has directivity in the sound collection range. In the present embodiment, twelve microphones 420a to 420l that can collect sound within a radial range of 30 degrees on the left and right respectively are combined to constitute the microphone array A40. Therefore, it can be said that the microphone array A40 has a sound collection range around 360 ° (omnidirectional).

  The microphone array A40 has a sound collection range in all directions. As described above, the microphone array A40 includes a plurality of microphones 420a to 420l and microphone amplifiers 430a to 430l that can be adjusted for each microphone. Therefore, the sound collection range can be controlled by adjusting the gain for each microphone.

  For example, in FIG. 4, the speaker's direction (angle 461) is specified with reference to the reference position mark 460, and the gain of the microphone 420e having a sound collection range in that direction is made higher than the gains of other microphones. Therefore, it is easier to collect a speaker's speech and suppress ambient noise.

  The arithmetic device 142, the main storage device 143, the communication device 144, and the sound collecting device 145 are connected to each other via a bus 141.

  The microphones 420a to 420l, the microphone amplifiers 430a to 430l, and the mixer 440 of the microphone array A40 described above are realized by the sound collecting device 145. The microphone gain control unit 410 and the speech encoding unit 450 of the microphone array A40 are realized by a program that causes the arithmetic unit 142 to perform processing. This program is stored in the main storage device 143 or a ROM device (not shown), loaded onto the main storage device 143 for execution, and executed by the arithmetic unit 142.

  The above is the hardware configuration example of the video conference system 1 in the present embodiment. However, the configuration is not limited to this, and other hardware may be used. For example, the network 70 may be a wireless network or a mobile phone network, and the conference terminal and the camera, microphone array, display, and speaker may be connectable by wireless communication such as Bluetooth (registered trademark).

  [Description of Operation] Next, the operation of the video conference system 1 in this embodiment will be described. FIG. 5 is a diagram illustrating an arrangement example of the camera A30 and the microphone array A40 of the video conference system 1 in the present embodiment. In FIG. 5, the microphone array A40 is arranged on the conference table (preferably near the center of the conference participant), and the camera A30 is arranged at one end of the conference table. At this time, the microphone array A40 is installed with the reference position mark 460 facing the front of the camera A30.

  Here, when the front direction of the camera A30 is set as a reference (0 degree), the right direction is a plus direction, and the left direction is a minus direction and expressed by the 360 degree method, the installation direction 501 of the microphone array A40 viewed from the camera A30 is minus ( -) It is assumed that the microphone array A40 is arranged at 5 °, that is, slightly to the left of the front of the camera A30. The reference position of the microphone array A40 is the position of the reference position mark 460. Further, it is assumed that the distance 500 from the camera A30 to the microphone array A40 is 2 m.

  The information is specified in an initialization process described later, and is stored in the microphone array position table 600 of the microphone array position data memory 230.

  Here, the initialization process will be described. First, the camera A30 zooms up the microphone array A40 as shown in FIG. 9 to photograph it as an initial setting image 340, and focuses on the reference position mark 460 of the microphone array A40. The operation is performed by a conference participant who uses the conference room A10 by appropriately operating the optical device 35.

  At that time, the control unit (not shown) of the conference terminal A20 displays the initial setting video 340 on the display A50 of the conference room A10. In the initialization process, the control unit superimposes and displays the scope 350 indicating the aiming reference at the center of the video. The conference participant adjusts the direction (pan direction) of the camera A30 so that the scope 350 overlaps the reference position mark 460 and focuses.

  The camera information notification unit 320 identifies the distance 500 between the camera A30 and the microphone array A40 and the installation direction 501 based on the focal length when the subject is in focus, and notifies the conference terminal A20.

  The above is the flow of the initialization process. By the initialization process, the position of the microphone array A40 viewed from the camera A30 can be specified.

  Next, the microphone array control process during the conference related to the conference room A10 will be described with reference to FIG. In the microphone array control process, the sound around the subject projected by the camera is collected with high quality.

  First, the camera A30 performs a pan / zoom operation (step S001). Specifically, the camera operation enable / disable control unit 330 of the camera A30 operates the conference terminal A20 by a conference participant in the conference room A10, or operates another conference terminal B21 or conference terminal C22 participating in the conference. A pan / zoom change instruction to the camera A30 according to the above is accepted. Then, when the camera operation is valid, the pan / zoom control unit 310 controls the optical device 35 so as to realize the pan / zoom operation according to the received change instruction. If the camera operation is invalid, the camera operation valid / invalid control unit 330 discards the received change instruction and does not respond.

  Next, the camera information notification unit 320 determines whether the camera operation has not been performed for a specified time (step S002). Specifically, the camera information notification unit 320 determines whether or not a predetermined time (for example, 3 seconds) has elapsed after the last pan / zoom operation in step S001. When the camera operation is performed within the specified time, that is, when the predetermined time has not passed (“No” in step S002), the camera information notification unit 320 performs step S002 again.

  When the camera operation is not performed within the specified time, that is, when a predetermined time has passed (“Yes” in step S002), the camera information notification unit 320 specifies the pan direction of the camera. The conference terminal A20 is notified (step S003), the distance between the camera and the subject is specified and notified to the conference terminal A20 (step S004), the angle of view captured by the camera is specified, and the conference terminal A20 is notified. (Step S005).

  As described above, in this embodiment, the camera information notification unit 320 specifies the pan direction from the difference between the orientation of the pedestal (front) and the orientation of the optical axis of the camera, and the distance to the subject is predetermined. The angle of view is determined according to the amount of extension of the lens group.

  When the camera state detection unit 210 of the conference terminal A20 receives the notification in steps S003 to S005, the camera operation enable / disable instruction unit 240 issues a camera operation lock instruction to the camera A30 (step S006).

  Upon receiving the camera operation lock instruction in step S006, the camera operation enable / disable control unit 330 of the camera A30 sets the pan / zoom operation to be invalid (step S007).

  The microphone array control unit 220 of the conference terminal A20 performs a microphone array setting calculation process described later (step S008).

  Then, the microphone array control unit 220 issues a microphone gain setting instruction including the setting value of the microphone gain specified by the microphone array setting calculation process to the microphone gain control unit 410 of the microphone array A40 (step S009).

  The microphone gain controller 410 of the microphone array A40 performs a microphone gain setting process (step S010). Specifically, microphone gain control unit 410 sets a microphone gain for each of microphone amplifiers 430a to 430l in accordance with the microphone gain setting instruction instructed in step S009.

  The camera operation enable / disable instruction unit 240 issues a camera operation lock release instruction to the camera A30 (step S011).

  When the camera operation enable / disable control unit 330 of the camera A30 receives a camera operation lock release instruction in step S011, it performs a camera operation lock release process (step S012). Specifically, the camera operation valid / invalid control unit 330 shifts to a state where the pan / zoom operation is effectively accepted.

  The above is the flow of the microphone array control process. According to the microphone array control process, the distance between the position of the image acquisition unit capable of panning left and right and the position of the subject reflected in the image acquired by the image acquisition unit, the pan direction of the image acquisition unit, Using the distance between the position of the image acquisition means and the position of the sound collection means capable of changing the sound collection range, and the direction of the position of the sound collection means with respect to the position of the image acquisition means. The sound collection range can be changed. That is, in the state where the camera and the microphone are arranged independently, the microphone sensitivity can be increased in the direction of the subject, so that it can be said that the quality of sound collection can be improved.

  Next, the microphone array setting calculation process performed in step S008 of the microphone array control process and its principle will be described with reference to FIG. In this description, it is assumed that the conference is held in the arrangement as shown in FIG. That is, in the state where the camera A30 and the microphone array A40 shown in FIG. 5 are arranged, the optical axis direction (pan angle 503) of the camera A30 directed toward the speaker and the direction of the microphone array A40 viewed from the camera A30 The difference between (installation direction 501) and the difference angle α504. That is, the pan angle 503 from the front of the camera A30 is an angle obtained by combining the installation direction 501 and the difference angle α504.

  The distance between the microphone array A40 and the subject is a distance a, the distance between the subject and the camera A30 is a distance b502, and the distance between the camera A30 and the microphone array A40 is c500. In addition, it is assumed that the angle of view 505 captured by the camera A30 is an angle of view that can be captured sufficiently by the speaker in the field of view.

  Further, the difference angle between the direction of the microphone array A40 viewed from the speaker and the direction of the camera A30 is defined as a difference angle γ, and the difference angle between the direction of the speaker viewed from the microphone array A40 and the direction of the camera A30 Is the difference angle β.

  In the case of the arrangement as described above, the relationship between the distance a, the distance b502, the distance c500, the difference angle α504, the difference angle β, and the difference angle γ is represented by the cosine theorem 90 (Equation 1, Equation 2, and Equation 3 below). Is established.

  That is, it can be said that the above relationship is established. Therefore, the microphone array control unit 220 identifies the direction in which the microphone array A40 collects sound (that is, the difference angle β) by solving the equations (1), (2), and (3).

  In this embodiment, the microphone array control unit 220 receives the pan direction in step S003, the distance from the camera A30 to the subject in step S004, and the angle of view information in step S005. The distance 601 between the camera A and the microphone array A and the reference pan direction angle 602 in the microphone array position table 600 stored in FIG. The microphone array control unit 220 identifies the difference angle α 504 from the difference between the pan direction and the reference pan direction angle 602, identifies the distance from the camera A 30 to the subject as the distance b 502, and the distance 601 between the camera A and the microphone array A. Is determined as the distance c500, and the difference angle β is calculated by substituting it into each equation of the cosine theorem 90.

  Then, the microphone array control unit 220 identifies the microphone 420 including the direction of the difference angle β as the microphone in the sound collection direction, and also identifies the microphone 420 adjacent to the microphone. Then, the output level (microphone gain setting value) of the microphone amplifier 430 is specified using the field angle information received in step S005 and the information in the microphone array output level table 650. Note that the sensitivity (gain) of the microphone in the sound collection direction is set higher than the sensitivity of any other microphone, and the sensitivity (gain) of the microphone other than the microphone in the sound collection direction and the microphone adjacent to the microphone in the sound collection direction is set. By setting the sensitivity lower than the sensitivity of any other microphone, the speech content of the speaker can be obtained with higher quality.

  The above is the principle and processing content of the microphone array setting calculation processing.

  The video conference system 1 including the microphone control device to which the first embodiment according to the present invention is applied has been described above. According to the conference terminals A20, B21, and C22 that are microphone control devices to which the first embodiment is applied, the microphone sensitivity can be increased in the direction of the subject in a state where the camera and the microphone are arranged independently. It can be said that the quality of sound collection can be improved.

  The present invention is not limited to the first embodiment. The first embodiment described above can be variously modified within the scope of the technical idea of the present invention. For example, in the first embodiment, the conference terminal A20 specifies the output level of the microphone amplifier in the sound collection direction according to the angle of view. However, the present invention is not limited to this, and the output level may be adjusted according to the distance from the microphone array to the speaker. For example, the output level adjustment table 700 shown in FIG. 11 may be stored in the microphone array position data memory 230, and the output level calculated in the microphone array setting calculation process may be adjusted. As a result, a more appropriate output level can be set, so that it can be said that an appropriate sound quality can be ensured.

  Specifically, the output level adjustment table 700 stores the output level 702 of the microphone selected as the sound collection direction in association with the distance 701 from the microphone array to the speaker. When the microphone array control unit 220 calculates the distance from the microphone array A40 to the speaker, that is, the distance a by the cosine theorem 90, the microphone array control unit 220 acquires the output level of the microphone according to the distance a using the output level adjustment table 700. Note that the microphone array control unit 220 sets the output level within a range equal to or smaller than the output level of the microphone in the sound collection direction specified using the microphone array output level table 650 and the angle of view. Then, the microphone array control unit 220 issues a microphone gain setting instruction at the acquired output level.

  That is, according to the modified embodiment, it can be said that not only the sound collection direction and sound collection range of the microphone array A40 but also the background noise of the speaker can be reduced.

  For example, as shown in FIG. 12, it is assumed that the sound collection possible range 520 of the microphone array A40 is sufficiently wide with respect to the seating position of the conference participant. In this case, the sound collection range of the microphone array A40 after the sound collection range control by the microphone control device in the first embodiment is the sound collection range 521 after the control. At this time, the sound collection range of the microphone array A40 is reduced to the sound collection range 522 by reducing the output level of the microphone 420 selected as the sound collection direction using the table shown in FIG. That is, the noise generated behind the speaker can be reduced.

  In the embodiments described so far, the microphone array A40 is arranged on the conference table and the camera A30 is arranged on the end of the conference table. However, the microphone array A40 is within the sound collection range of the microphone array A40 and the shooting range of the camera A30. As long as there are conference participants at positions where the camera A30 can capture images, they can be placed anywhere. Further, the functions of the respective units in the conference terminal A20 described in the present embodiment may be distributed and incorporated in the camera A30 or the microphone array A40, and processing may be performed in cooperation between the camera A30 and the microphone array A40. . Furthermore, the present invention can be similarly applied not only to a video conference system but also to a surveillance system using a network surveillance camera.

  In addition, the above-described modification of the embodiment may be applied alone, or may be applied in a partial combination.

  In the above, this invention was demonstrated centering on embodiment. In the above embodiment, communication such as the Internet is assumed to be communication using an existing protocol such as HTTP, FTP, SIP, POP, SMTP, TCP / IP, etc. It may be.

DESCRIPTION OF SYMBOLS 1 ... Video conference system, 10 ... Conference room A, 11 ... Conference room B, 12 ... Conference room C, 20 ... Conference terminal A, 21 ... Conference terminal B, 22. ..Conference terminal C, 30 ... Camera A, 31 ... Camera B, 32 ... Camera C, 40 ... Microphone array A, 41 ... Microphone array B, 42 ... Microphone array C 50 ... Display A, 51 ... Display B, 52 ... Display C, 60 ... Speaker A, 61 ... Speaker B, 62 ... Speaker C, 70 ... Network, 80 ... Server, 90 ... Cosine theorem, 340 ... Initial setting video, 350 ... Scope, 420a-420l ... Microphone, 430a-430l ... Microphone amplifier, 500 ... Distance c, 501 ... How to install 502 b Distance 503 Pan angle 504 Difference angle α 505 Angle of view 520 Sound collection range 521 Sound collection range 522 Sound collection range, 600: microphone array position table, 650: microphone array output level table, 700: output level adjustment table

Claims (13)

Image acquisition means that can pan left and right;
A control device for controlling sound collection means capable of changing a sound collection range,
The distance between the position of the image acquisition means and the position of the subject reflected in the image acquired by the image acquisition means, the pan direction of the image acquisition means, the position of the image acquisition means, and the sound collection means A sound collection range changing means for changing a sound collection range of the sound collecting means using a distance between the sound collecting means and a direction of the position of the sound collecting means with reference to the position of the image acquisition means. ,
A control device characterized by that. The control device according to claim 1,
The sound collection range changing means includes
Identify the direction of the subject as seen from the position of the sound collection means and change the sound collection range in that direction;
A control device characterized by that. The control device according to claim 1 or 2,
The sound collection range changing means includes
Obtaining a distance between the position of the sound collecting means and the subject, and adjusting the gain of the sound collecting means according to the distance;
A control device characterized by that. The control device according to any one of claims 1 to 3,
The sound collection means includes a plurality of microphones having a predetermined sound collection range,
The sound collection range changing means changes the sound collection range by adjusting a gain for each microphone.
A control device characterized by that. The control device according to claim 4,
The sound collection range changing means sets a gain of a microphone that collects the direction of the subject higher than a gain of a microphone that collects the sound in another direction.
A control device characterized by that. The control device according to any one of claims 1 to 5,
The sound collection range changing means acquires information for specifying an angle of view from the image acquisition means, and changes the sound collection range according to the acquired angle of view.
A control device characterized by that. The control device according to claim 6,
The sound collection range changing means collects sound by increasing the sensitivity of the sound collection range as the angle of view is narrower.
A control device characterized by that. The control device according to any one of claims 1 to 7,
The sound collection range changing means calculates a difference with the pan direction of the image acquisition means based on the direction of the position of the sound collection means with respect to the position of the image acquisition means, and uses the cosine theorem to calculate the subject. Identify the direction of the and change the sound collection range,
A control device characterized by that. The control device according to any one of claims 1 to 8,
The sound collection range changing means changes the sound collection range when the pan operation of the image acquisition means is not performed for a predetermined time.
A control device characterized by that. The control device according to any one of claims 1 to 9,
The sound collection range changing means invalidates a pan operation of the image acquisition means while changing the sound collection range;
A control device characterized by that. An imaging device comprising a left and right panning means;
A sound collecting device comprising means for changing the sound collecting range;
A control device for controlling the imaging device and the sound collecting device,
The controller is
The distance between the position of the imaging device and the position of the subject reflected in the video acquired by the imaging device, the pan direction of the imaging device, the position of the imaging device, and the position of the sound collector A sound collection range change instructing unit that instructs to change the sound collection range of the sound collection device using a distance between the sound collection device and a direction of the sound collection device position with respect to the position of the imaging device.
A control system characterized by that. Panning means for panning left and right;
An angle-of-view changing operation means for changing the angle of view being shot;
Pan direction information acquisition means for acquiring pan direction information for specifying the pan direction;
Distance information acquisition means for acquiring distance information for specifying the distance to the subject;
Angle-of-view information acquisition means for acquiring angle-of-view information for specifying the angle of view being shot;
An output means for outputting the pan direction information, the distance information, and the angle of view information;
The output means performs the output when a predetermined time elapses after completion of both the pan operation means and the angle of view change operation means.
An imaging apparatus characterized by that. Image acquisition means that can pan left and right;
A sound collection means capable of changing a sound collection range, and a control method of a control device for controlling the sound collection range,
The controller is
The distance between the position of the image acquisition means and the position of the subject reflected in the image acquired by the image acquisition means, the pan direction of the image acquisition means, the position of the image acquisition means, and the sound collection means A sound collection range changing step of changing the sound collection range of the sound collecting means using the distance between the sound collecting means and the direction of the position of the sound collecting means relative to the position of the image acquisition means To
A control method characterized by that.

Images