JP2000101901A - Image pickup device and method therefor, image pickup device control system and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To quickly perform the focus control in the desired area of a photographing image by making use of the information on sounds or voices which are generated by a subject for image pickup. SOLUTION: This device includes an image pickup means 201 which photographs an object, plural microphones 109 constructing a sound source detection means which detects sounds generated from the photographing object that is to be photographed by the means 201, a sound source position estimation means 108 which estimates a sound source position, based on the sound source signals outputted from the microphones 109 and an automatic focus adjustment means 103 which focuses on the sound source position of the object, based on the information on the sound source position estimated by the means 108. Thus, it is possible to quickly focus on a desired position without panning nor tilting a camera, even if there is no desired object for focusing at the center of a photographing object.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an imaging apparatus, an imaging method, an imaging apparatus control system, and a storage medium.
The present invention relates to a technique for operating an imaging device installed in a remote place in accordance with a control command from a computer.

[0002]

2. Description of the Related Art Conventionally, a camera and a computer installed in a remote place are connected with an RS serial cable or a computer.
A remote control system for a camera has been developed in which the camera is connected via a network or the like, a control command is transmitted from a computer terminal, and the camera is panned and tilted to control an imaging direction or to control a zoom magnification. .

[0003] In such a remote control system,
Software for operating the camera is incorporated in the computer terminal, and the camera can be operated from the computer terminal side by transmitting a control command to the camera by operating a keyboard or a mouse.

Further, when the connected camera has an automatic focusing (AF) function or the like, the computer can also execute such a control command. In addition, a technique for estimating the position of a sound source using a microphone array has been developed.

[0005]

However, in the conventional camera control system, the focus, exposure, and the like are usually adjusted to the center of the subject to be photographed. Therefore, in order to focus and expose an object other than near the center of the subject to be photographed, the camera is moved by panning and tilting so that the object to be photographed is located at the center of the camera, and the detection area in the camera is set. It was necessary to adjust the position of the object to be photographed, which was very troublesome.

SUMMARY OF THE INVENTION It is an object of the present invention to quickly execute control for focusing on a desired area of a photographed image by using information of sound or sound generated from a subject to be imaged.

[0007]

According to the present invention, there is provided an image pickup apparatus comprising: an image pickup means for picking up an image of a subject; a sound source detecting means for detecting sounds generated from the subject picked up by the image pickup means by a plurality of microphones; A sound source position estimating means for estimating a sound source position based on a sound source signal output from a sound source detecting means; and a sound source position generated from within the subject based on sound source position information estimated by the sound source position estimating means. And automatic focus adjustment means for adjusting Further, other features of the imaging apparatus of the present invention include: an imaging unit that captures an image of a subject; a sound source detection unit that detects, using a plurality of microphones, sounds generated from the subject captured by the imaging unit; Sound source position estimating means for estimating a sound source position based on a sound source signal output from a sound source detecting means, detection area managing means for managing a detection area in an image signal output from the imaging means, and sound source position estimating means A first focus adjustment function for focusing on a sound source position in the subject based on the information on the sound source position estimated by the above, and a focus adjustment based on an image signal of a detection area managed by the detection area management means. Automatic focus adjustment means having a second focus adjustment function. Another feature of the imaging apparatus of the present invention is that the imaging apparatus further includes an imaging parameter output unit that outputs information on an imaging parameter including a current exposure value of the imaging unit and a subject distance to the outside. I have.

An imaging device control system according to the present invention is an imaging device control system for controlling the operation of an imaging device from a remote control device installed at a remote place. The imaging device includes: an imaging unit for imaging a subject; Sound source detecting means for detecting sound generated from a subject imaged by a plurality of microphones; sound source position estimating means for estimating a sound source position based on a sound source signal output from the sound source detecting means; And an automatic focus adjusting means for focusing on a sound source position generated from within the subject based on the sound source position information estimated by the means. Another feature of the imaging device control system of the present invention is that in an imaging device control system that controls the operation of the imaging device from a remote control device installed at a remote location, the imaging device includes an imaging device that captures an image of a subject. Means, sound source detecting means for detecting, using a plurality of microphones, sound generated from a subject imaged by the imaging means, and sound source position estimating means for estimating a sound source position based on a sound source signal output from the sound source detecting means A detection area management means for managing a detection area in an image signal output from the imaging means; and a sound source position in the subject based on information on the sound source position estimated by the sound source position estimation means. A first focus adjustment function and a second focus adjustment for performing focus adjustment based on the image signal of the detection area managed by the detection area management means. And an automatic focusing means having an ability, the remote control device, the first focusing functions of the above-described automatic focusing means and the second
And a focus adjustment function selecting means for selecting one of the focus adjustment functions. Other features of the imaging device control system of the present invention include:
The remote control device includes a display unit that displays the estimated sound source position information output from the imaging device or a detection area in an image signal output from the imaging unit so as to overlap a captured image. . Another feature of the imaging device control system of the present invention is that the imaging device includes an imaging parameter output unit that outputs information on a current imaging parameter to the remote control device.

An imaging method according to the present invention is an imaging method in which the operation of the imaging device is controlled from a remote control device installed at a remote place. Sound source detection processing for detecting sound generated from a plurality of microphones, sound source position estimation processing for estimating a sound source position based on a sound source signal obtained by the sound source detection processing, and sound source estimated by the sound source position estimation processing Based on location information,
The automatic focus adjustment processing for focusing on a sound source position generated from within the subject is performed. According to another feature of the imaging method of the present invention, in an imaging method in which the operation of the imaging device is controlled from a remote control device installed in a remote place, the imaging device includes an imaging process of imaging a subject; A detection process of detecting sounds generated from a subject imaged by the imaging process by a plurality of microphones, a sound source position estimation process of estimating a sound source position based on a sound source signal obtained by the detection process, and a sound source position estimation process obtained by the imaging process. Detection area management processing for managing a detection area in an image signal to be obtained, first focus adjustment processing for focusing on a sound source position in the subject based on information on the sound source position estimated by the sound source position estimation processing, Performing a second focus adjustment process for performing focus adjustment based on the image signal of the detection region managed by the detection region management process, wherein the remote control device A first focus adjustment process for focusing on a sound source position in the subject based on information on a sound source position estimated by the sound source position estimation process, and a first focus adjustment process based on an image signal of a detection region managed by the detection region management process. And a second focus adjustment process for performing focus adjustment by performing an adjustment function selection process. According to another feature of the imaging method of the present invention, the remote control device estimates the sound source position information output from the imaging device or a detection area in an image signal obtained by the imaging process. It is characterized in that display processing for superimposing and displaying is performed. Another feature of the imaging method of the present invention is that the imaging device performs an imaging parameter output process of outputting information on a current imaging parameter to the remote control device.

[0010] A storage medium according to the present invention is characterized in that a program for causing a computer to function as each means described above is stored so as to be readable from the computer. Another feature of the storage medium of the present invention is that the steps constituting the imaging method are stored so as to be readable from a computer.

[0011]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an imaging apparatus, an imaging method, an imaging apparatus control system, and a storage medium according to the present invention will be described. FIG. 1 is a block diagram showing an embodiment of the present invention and showing a schematic configuration of a camera control system.

In FIG. 1, reference numeral 100 denotes a camera, which is mainly composed of a video input unit 101 and a microphone array 109 which will be described below.
It is constituted by. Reference numeral 200 denotes a computer terminal (hereinafter, referred to as a client), which mainly includes a video input interface 201 to an external storage device 208 and the like.

An image input unit 101 includes a lens unit including a lens for performing a focus adjustment and the like, a diaphragm for adjusting a light amount, and a CCD for converting an optical image incident through the lens into an electric signal. It is composed of

The AE circuit 102 controls exposure such as shutter speed, aperture, and AGC based on an image signal obtained from the CCD of the video input unit 101. AF circuit 10
3 extracts the high-frequency component of the image signal obtained from the CCD and controls the lens to be driven to the portion having the highest sharpness, or controls the lens according to the sound source position information obtained by the sound source position estimating unit 108. Control to drive.

The detection area management unit 104 manages the position of the detection area with respect to the captured image for executing automatic exposure adjustment AE or automatic focus adjustment AF. Detection area management unit 104
Is to move the detection area according to a command from the client 200.

The video output I / F 105 is connected to the video input I / F 201 on the client 200 side, and outputs image data subjected to predetermined processing such as compression to the client. The command I / F 106 is the same as the command I / F 202 on the client 200 side.
Client 200 connected by RS232C etc.
And the transmission and reception of control commands or camera parameter information.

The CPU 107 controls the entire camera 100, and causes each unit to execute an appropriate process according to a control command from the client 200. Sound source position estimation unit 108
Estimates the position of the sound source based on the acoustic signal input from the microphone array 109. Microphone array 10
The direction and distance can be obtained as the sound source position information from 9, and the distance information is used for the automatic focus adjustment of the AF circuit 103.

The microphone array 109 is provided for obtaining an acoustic signal necessary for estimating a sound source position. In the present embodiment, the microphone array 109 is constituted by an aggregate of a plurality of microphones.

On the other hand, a video input I / F 201 provided in the client 200 is for receiving image data from the camera 100. Command I / F 202 is for camera 100
The display 203 is for displaying an image captured by a camera based on the received image data, and is composed of a CRT or an LCD.

The input unit 204 includes a keyboard or a mouse functioning as a pointing device, and inputs a control command. CPU 205 is a client
The CPU 200 controls the entire operation, reads a program stored in the ROM 206 into the RAM 207, and executes each operation process based on the read program.

The ROM 206 stores various programs for executing processing of image data input from the input unit 204 or processing of the CPU 205. The RAM 207 provides a work area for various programs or a temporary save area for image data input from the input unit 204.

The external storage device 208 is constituted by an FDD, a hard disk, a CD-ROM, or the like, and is capable of storing a program for executing a client process in these storage devices. The processing can be executed by reading.

Next, operation processing of the camera control system according to the present embodiment will be described. FIG. 2 shows the camera 100
It is an operation processing flowchart regarding AF processing.

As shown in FIG. 2, first, in the first step S101, it is checked whether or not an AF switching command has been received from the client 200. If the switching command has not been input, step S103 is performed as it is.
Proceed to.

On the other hand, if the result of determination in step S101 is that a switch command has been input, the AF flag is set in step S102. In this case, when a command for executing the focus adjustment based on the image signal of the detection area is input, “0” is set as the AF flag. If a command for executing focus adjustment based on the estimated sound source position is input, “1” is set as the AF flag.

Next, in step S103, it is analyzed whether or not the command received from the client 200 is a detection area movement command. As a result of this analysis, if it is determined that the received command is a detection area movement command,
In step S104, the CPU 107
The detection area is moved for each pixel on the captured image.

If the received command is not a detection area movement command, the flow returns to step S101 to wait until a command is input.

Next, in step S105, the AE circuit 102
AE control is performed, and then the process proceeds to step S106. In step S106, referring to the AF flag set in step S102, if the set value is "0",
Proceed to S107, and if the set value is “1”, step S108
Proceed to.

In step S107, the image signal of the moved detection area is extracted, and the AF circuit 103 performs focus adjustment based on the image of the detection area.

On the other hand, in step S108, focus adjustment is performed based on the estimated sound source position. The sound source position is estimated by the sound source position estimating unit 108 using the acoustic signal input from the microphone array 109. This method of estimating the sound source location is described in the literature "Minor Special Issue Microphone Array,
84-414 (1995) "and the literature" Microphone array technology for speech recognition under noise, Acoustic Journal 53, 872-876 (1997) ".

The information on the distance and direction can be obtained by estimating the position of the sound source.
Adjust the focus with. When sound source position information cannot be obtained, such as when there is no sound, message data indicating that sound source position information cannot be obtained is created. This message is sent to the camera client 200 together with other data in step S111.
Send to

Next, in step S109, the AE circuit 102
Reference is made to camera parameter values such as an exposure value and a subject distance which are managed by the AF circuit 103, respectively. Then, in step S110, message data to which the value of the camera parameter is added together with the current position of the detection area is created. Then, in step S111, the message data is transmitted to the camera client 200.

(Other Embodiments) (1) In the configuration of the camera control system of FIG. 1, an example is shown in which the camera 100 and the client 200 are connected one-to-one, but as shown in FIG. A plurality of cameras and a plurality of clients may be connected via a network.
In this case, the specifications of the cameras may be the same or different.

(2) In the first embodiment, it is assumed that there is one sound source, but a plurality of sound sources may be used. In the case of a plurality of sound sources, it is necessary to determine where to focus, but the following methods are conceivable.

That is, the first method is a method in which priorities are set according to the magnitude relationship between sounds, and the sound is focused on the larger or smaller sound.

The second method is a method of prioritizing according to the incident angle of the sound and focusing on the smaller or larger incident angle.

In the third method, information on sound source candidates is transmitted to the client 200 without prioritizing. On the client 200 side, this is a method in which a sound source position is marked by superimposing it on an image, and the user selects one of the candidate sound sources by a command operation.

(3) In the first embodiment, when a sound source is not found, only a message to the effect that no sound source is found is transmitted to the client 200. However, when no sound source is found, the focus adjustment method is detected. A method of automatically switching to a method based on the image of the area may be used.

In this case, a message to the effect that the focus adjustment method has been changed is transmitted to the client 200. Conversely,
In the case where the focus is adjusted by the method based on the image of the detection area, if a sound is detected, the method may be automatically switched to the method based on the estimated sound source position.

(4) In the first embodiment, since the focus is adjusted using only the distance information, if the object to be focused is not in front of the camera, a slight error occurs in the distance. An embodiment for correcting this error will be described below, and a method for performing AE control will be described.

First, in the first embodiment,
In S108, the focus adjustment is performed using only the distance information. In the present embodiment, both the distance and direction information obtained from the sound source position estimating unit 108 are used.

FIG. 4 shows the obtained information. In FIG. 4, "X" is the sound source position, h is the distance information obtained from the sound source position estimating unit 108, and the angle θ is the direction information. In the first embodiment, focus adjustment is performed only with the distance information h.
In this embodiment, “distance information h × cos θ” is calculated by the sound source position estimating unit 108, and the AF circuit 103 is calculated using the calculated value.
Focus adjustment.

Further, the detection area management unit 104 moves the detection area in the angle θ direction for each pixel on the captured image.
AE circuit 102 based on the image signal of the newly obtained detection area
Executes AE control.

(Other Embodiments of the Present Invention) The present invention is applied to a system constituted by a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), and is applied to an apparatus constituted by one device. May be applied.

Also, the functions of the above-described embodiments are implemented in an apparatus connected to the above-mentioned various devices or a computer in a system so that various devices are operated so as to realize the functions of the above-described embodiments. Supplies the software program code for performing the
The invention implemented by operating the various devices according to the program stored in U) is also included in the scope of the present invention.

In this case, the program code of the software implements the functions of the above-described embodiment, and the program code itself and means for supplying the program code to a computer are provided.
For example, a storage medium storing such a program code constitutes the present invention. As a storage medium for storing such a program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM,
A magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the supplied program code, not only the functions of the above-described embodiment are realized, but also the OS (operating system) or other operating system running the program code on the computer. It goes without saying that such program codes are also included in the embodiment of the present invention when the functions of the above-described embodiment are realized in cooperation with the application software or the like.

Further, after the supplied program code is stored in a memory provided in a function expansion board of a computer or a function expansion unit connected to the computer, the function expansion board or the function expansion unit is specified based on the instruction of the program code. It is needless to say that the present invention also includes a case where a CPU or the like provided for performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0049]

As described above, according to the present invention,
A plurality of microphones detect sound generated from a subject imaged by an imaging device to obtain a sound source signal, estimate a sound source position based on the sound source signal, and, based on the estimated sound source position information, Focusing on the sound source position generated from inside, even if there is no object to focus on the center of the subject to be photographed, it is possible to perform focus adjustment based on sound or audio information, It is possible to quickly focus on a desired position without performing the pan and tilt operations of the camera.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an embodiment of the present invention and schematically illustrating a configuration of a camera control system.

FIG. 2 is a flowchart showing a processing procedure of the present invention.

FIG. 3 is a diagram showing the concept of a camera control system when connected via a network.

FIG. 4 is an explanatory diagram of distance correction.

[Explanation of symbols]

 100 Camera 101 Video input unit 102 AE circuit 103 AF circuit 104 Detection area management unit 105 Video output I / F 106 Command I / F 107 CPU 108 Sound source position estimation unit 109 Microphone array 200 Client 201 Video input I / F 202 Command I / F F 203 Display 204 Input unit 205 CPU 206 ROM 207 RAM 208 External storage device

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C022 AB23 AB29 AB62 AB65 AC00 AC03 AC27 AC69 AC72 5C054 AA02 CC02 CF05 CH05 DA09 FA01 FA09 FC12 HA24 HA25 5C064 AA01 AA02 AC04 AC06 AC07 AC09 AD08 AD09 AD14

Claims (13)

[Claims] 1. An image pickup means for picking up an image of a subject, a sound source detecting means for detecting, using a plurality of microphones, a sound generated from the subject picked up by the image pickup means, and a sound source signal output from the sound source detecting means. Sound source position estimating means for estimating the sound source position, and automatic focus adjusting means for focusing on a sound source position generated from within the subject based on the sound source position information estimated by the sound source position estimating means. Characteristic imaging device. 2. An image pickup means for picking up an image of a subject, a sound source detecting means for detecting, using a plurality of microphones, a sound generated from the subject picked up by the image pickup means, and a sound source signal outputted from the sound source detecting means. Sound source position estimating means for estimating a sound source position, a detection area managing means for managing a detection area in an image signal outputted from the imaging means, and information on a sound source position estimated by the sound source position estimating means. Automatic focus adjustment having a first focus adjustment function for focusing on a sound source position in the subject and a second focus adjustment function for performing focus adjustment based on an image signal of a detection area managed by the detection area management means An imaging apparatus comprising: 3. The imaging apparatus according to claim 1, further comprising an imaging parameter output unit configured to output information regarding an imaging parameter including a current exposure value of the imaging unit and a subject distance to the outside. . 4. An imaging device control system for controlling the operation of an imaging device from a remote control device installed at a remote location, wherein the imaging device is configured to generate an image of an object, and to generate an image from the object imaged by the imaging device. Sound source detecting means for detecting a sound to be emitted by a plurality of microphones, sound source position estimating means for estimating a sound source position based on a sound source signal output from the sound source detecting means, and a sound source position estimated by the sound source position estimating means An image pickup apparatus control system comprising: an automatic focus adjustment unit that focuses on a sound source position generated from within the subject based on information. 5. An imaging device control system for controlling the operation of an imaging device from a remote control device installed at a remote location, wherein the imaging device is configured to generate an image of an object and an image generated by the object imaged by the imaging device. Sound source detecting means for detecting sound to be emitted by a plurality of microphones, sound source position estimating means for estimating a sound source position based on a sound source signal output from the sound source detecting means, and an image signal output from the imaging means. Detection area management means for managing a detection area, a first focus adjustment function for focusing on a sound source position in the subject based on information on the sound source position estimated by the sound source position estimation means, and the detection area management means Automatic focus adjustment means having a second focus adjustment function for performing focus adjustment based on the image signal of the detection area managed by Controller, an imaging device control system comprising: a focus adjustment function selecting means for selecting anyone one of said first focusing function and a second focus adjustment functions of the above-described automatic focusing means. 6. The remote control device includes display means for displaying estimated sound source position information output from the imaging device or a detection area in an image signal output from the imaging means so as to be superimposed on a captured image. The imaging device control system according to claim 4 or 5, wherein 7. The imaging device according to claim 4, wherein the imaging device includes an imaging parameter output unit that outputs information on a current imaging parameter to the remote control device. Control system. 8. An imaging method for controlling the operation of an imaging device from a remote control device installed at a remote location, wherein: an imaging process of imaging the subject by the imaging device; and a sound generated from the subject imaged by the imaging process. Based on sound source position information estimated by the sound source position estimation process for estimating the sound source position based on the sound source signal obtained by the sound source detection process. Performing an automatic focus adjustment process for focusing on a sound source position generated from within the subject. 9. An imaging method for controlling the operation of an imaging device from a remote control device installed at a remote place, wherein the imaging device is generated from an imaging process for imaging a subject and an object imaged by the imaging process. Detection processing for detecting sound with a plurality of microphones, sound source position estimation processing for estimating a sound source position based on the sound source signal obtained by the detection processing, and detection for managing a detection area in an image signal obtained by the imaging processing A region management process and a first process of focusing on a sound source position in the subject based on information on the sound source position estimated by the sound source position estimation process.
And a second focus adjustment process of performing focus adjustment based on the image signal of the detection area managed by the detection area management process. The remote control device estimates by the sound source position estimation process. A first focus adjustment process for focusing on a sound source position in the subject based on the information on the sound source position thus performed, and a second focus adjustment based on an image signal of a detection region managed by the detection region management process. An imaging method characterized by performing an adjustment function selection process for selecting a focus adjustment process. 10. The remote control device according to claim 1, wherein the remote control device performs a display process of displaying the estimated sound source position information output from the imaging device or a detection area in an image signal obtained by the imaging process so as to be superimposed on a captured image. The imaging method according to claim 8, wherein: 11. The imaging device according to claim 8, wherein the imaging device performs an imaging parameter output process of outputting information on a current imaging parameter to the remote control device.
The imaging method according to claim 1. 12. A storage medium storing a program for causing a computer to function as each of the means according to claim 1 so as to be readable from the computer. 13. A storage medium, wherein the steps constituting the imaging method according to claim 8 are stored so as to be readable from a computer.