JP2023035585A - Imaging apparatus, method for controlling imaging apparatus, and program - Google Patents

Abstract

To achieve reproduction of realistic sound.SOLUTION: An imaging apparatus has: imaging means; and connection means that is connected with a first microphone and a second microphone. The imaging apparatus adjusts the gain for a voice signal input from the first microphone based on the amount of defocus of an image picked up by the imaging means.SELECTED DRAWING: Figure 1

Description

The present invention relates to an imaging device that acquires sounds from a plurality of microphones.

2. Description of the Related Art In recent years, imaging apparatuses have become widespread that can record the voice of a subject, environmental sounds, and the like together with an image by collecting sound with a plurality of microphones (hereinafter referred to as microphones) at the same time as the image is captured. Further, Japanese Patent Application Laid-Open No. 2002-200001 discloses a technique for emphasizing sound from a focused direction when photographing.

JP 2012-15651 A

If the sounds picked up by a plurality of microphones can be individually adjusted so as to match the captured image, it will be possible to reproduce the sound with a sense of realism with respect to the captured image. Here, if the technique disclosed in Patent Document 1 is used, it is possible to reproduce sound that emphasizes the left and right sides or the upper and lower sides of the subject. Reproduction is difficult.

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to make it possible to reproduce sound with a sense of reality.

The imaging device of the present invention includes an imaging means, a first microphone, a connection means for connecting a second microphone, and a defocus amount of an image captured by the imaging means from the first microphone. and adjusting means for adjusting the gain for the input audio signal.

According to the present invention, it is possible to reproduce realistic sounds.

It is a figure which shows the structural example of an imaging device. 4 is a control flowchart of the imaging device; It is a figure which shows an example of imaging|photography using an imaging device. It is a figure which shows an example of the image image|photographed with the imaging device. FIG. 10 is a diagram showing an example of a case in which a subject moves during shooting; FIG. 11 is a control flowchart of an imaging device according to a fourth embodiment; FIG. FIG. 11 is a diagram showing an example of photographing using an imaging device according to a fourth embodiment; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments according to the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all combinations of features described in the embodiments are essential for the solution of the present invention. The configuration of the embodiment can be appropriately modified or changed according to the specifications of the device to which the present invention is applied and various conditions (use conditions, use environment, etc.). Also, a part of each embodiment described later may be appropriately combined. In each of the following embodiments, the same configurations are given the same reference numerals.

In the present embodiment, an imaging apparatus capable of recording the voice of the main subject to be shot, the surrounding environmental sounds, etc. by collecting sounds with a plurality of microphones at the same time as shooting will be described as an example. In the present embodiment, the multiple microphones include the built-in microphone of the imaging device and the microphone near the subject (the microphone held by the person of the subject), but this is just an example. more microphones may be included, such as

Here, if each sound picked up by a plurality of microphones can be adjusted to a sound suitable for a captured image, it is possible to reproduce sound with a sense of realism with respect to the captured image. For example, if the volume of the sound picked up by the microphone near the subject and the sound picked up by the camera's built-in microphone can be appropriately adjusted according to the distance to the subject, sound reproduction with a sense of realism can be achieved. It is considered possible. However, in general, it is complicated and time-consuming to adjust the sound so that it matches the captured image, for example, to adjust the volume appropriately for the sound of the main subject being photographed and the surrounding environmental sound. This is such work, and it is a heavy burden on the user. In addition, when a subject moves while shooting a moving image, the sound volume is often not optimal for the image of the moving subject. In this case, after the video and audio are recorded, the user has to adjust the volume by adjusting the sound of each microphone while watching the reproduced video. Furthermore, the more microphones are placed, the wider the spatial area of the reproduced sound becomes, but the volume adjustment for each microphone becomes more complicated, and the workload of adjusting the volume after shooting becomes even heavier. On the other hand, as in the technique of Patent Document 1, it is conceivable to emphasize the sound from the focused direction. However, it is difficult to reproduce sound with a sense of realism with respect to the captured image.

<First Embodiment>
Therefore, the imaging apparatus according to the first embodiment has the configuration described below in order to enable realization of sound reproduction with a sense of presence, and performs the processing described below to reproduce the captured image. On the other hand, it is possible to realize sound reproduction with a clear sense of perspective and presence. The imaging apparatus of the first embodiment separates the main subject, which is the object of photography, from the background other than the main subject from the photographed image, and defocus amount (out-of-focus amount) representing the defocus amount (out-of-focus amount) for each pixel coordinate of the photographed image. Based on the map, the degree of blurring of the background is determined. That is, when the main subject is in focus, the imaging device determines that the greater the defocus amount of the background, the greater the degree of blurring of the background. Here, the smaller the aperture value of the imaging device, the larger the background blur, the smaller the F number, the larger the background blur, and the longer the focal length, the larger the background blur. For this reason, the imaging apparatus of the present embodiment determines the gain for the audio data by adding the aperture value, the F number, and the focal length at the time of shooting to the defocus amount representing the degree of blurring of the background. The audio data after volume adjustment is recorded together with the captured image data. The imaging apparatus of this embodiment increases the gain for sound picked up by the built-in microphone of the imaging apparatus relative to the gain for sound picked up by an external microphone near the main subject as the defocus amount of the background increases. , and record the audio data after adjusting the volume based on the gain. In other words, the smaller the defocus amount of the background, the smaller the gain of the sound picked up by the built-in microphone relative to the gain of the sound picked up by the external microphone near the subject. Audio data after adjusting the volume based on the gain is recorded. Note that when the defocus amount of the background is larger than a predetermined threshold value, the imaging apparatus of the present embodiment sets the gain for the sound picked up by the internal microphone relative to the gain for the sound picked up by the external microphone near the main subject. can be significantly larger. In other words, when the defocus amount of the background is equal to or less than a predetermined threshold value, the imaging apparatus sets the gain for the sound picked up by the built-in microphone relative to the gain for the sound picked up by the external microphone near the main subject. can be made smaller.

FIG. 1 is a block diagram showing a configuration example of an imaging apparatus according to a first embodiment that implements the above.
The photographing optical system unit 102 includes a focus lens unit, a zoom lens unit, an aperture/shutter unit, and the like. The actuator control unit 103 includes a motor driver IC, and drives various actuators of the imaging optical system unit 102, such as a focus lens unit, a zoom lens unit, and an aperture/shutter unit. Various actuators are driven based on actuator drive instruction data generated by the central control unit 101, which will be described later.

The imaging unit 104 includes an imaging device such as a CMOS sensor or a CCD sensor, photoelectrically converts an optical image formed on the imaging device surface by the imaging optical system unit 102 into an imaging signal (electrical signal), and converts the imaging signal into an electrical signal. Output to the imaging processing unit 105 . The imaging processing unit 105 converts the imaging signal input from the imaging unit 104 into a digital image signal (image data) and outputs the digital image signal (image data) to the image processing unit 106 .

The image processing unit 106 performs image processing on the image data input from the imaging processing unit 105 according to the application. The image processing performed by the image processing unit 106 includes, for example, electronic anti-vibration processing by image clipping processing and rotation processing, subject detection processing for detecting a subject from an image, area ratio of a subject (person, etc.) in an image. , and the like are included. Since these image processes are existing processes, detailed description thereof will be omitted. The image processing unit 106 outputs image data after image processing to the central control unit 101 .

The sound pickup unit 108 is a sound acquisition unit that includes a built-in microphone (camera built-in microphone) of the imaging apparatus, converts sound captured by the built-in microphone into an electric signal, and further converts it into a digital sound signal (sound data). and output to the audio processing unit 107 . In the case of the first embodiment, the built-in microphone of the imaging device corresponds to the first microphone.

The external sound pickup device 200 is an external sound acquisition unit (external microphone) wirelessly connected to the imaging device, and transmits a radio signal including collected sound data. In the case of the first embodiment, the external microphone of the external sound pickup device 200 corresponds to the second microphone. The external sound pickup device 200 is, for example, a wireless microphone, and a wireless signal transmitted from the wireless microphone is received by the wireless unit 109 inside the imaging device 100 . Note that the external sound pickup device 200 may be, for example, a wired cable microphone having a phone plug. In this embodiment, a case where the external sound pickup device 200 is a wireless microphone will be described as an example.

The wireless unit 109 performs wireless communication of image data and the like in compliance with wireless standards such as Wifi (registered trademark) and BLE (Bluetooth LE, Bluetooth is a registered trademark). In the imaging apparatus of this embodiment, the wireless unit 109 receives wireless signals including audio data from the external sound pickup device 200 . Radio section 109 converts the radio signal into reception data and sends it to central control section 101 . At this time, the central control unit 101 separates the audio data collected by the external sound pickup device 200 from the received data input from the radio unit 109, and further converts it into audio data that can be processed by the audio processing unit 107. and output to the audio processing unit 107 .

Note that when the external sound pickup device 200 is a wired cable microphone, an analog audio signal picked up by the wired cable microphone is input to the imaging device 100 via the microphone jack of the imaging device 100 . An analog audio signal input to the imaging apparatus 100 is converted into a digital audio signal (audio data) by an analog-to-digital converter (not shown), and then input to the audio processing unit 107 .

The sound processing unit 107 performs sound processing on the sound data collected by the sound collection unit 108 and the external sound collection device 200 respectively. In the case of this embodiment, the audio processing unit 107 performs audio processing such as adjusting the volume based on the gain determined by the central control unit 101 as will be described later.

A central control unit 101 is a control unit including a CPU, a ROM, and a RAM, controls the entire imaging apparatus 100, and performs various kinds of arithmetic processing. Illustrations of the CPU, ROM, RAM, etc. are omitted. The ROM stores control programs, initial set values, and the like according to the present embodiment. Arithmetic processing in the central control unit 101 is realized by executing the control program according to the present embodiment. The CPU of the central control unit 101 reads the control program according to this embodiment from the ROM, develops it in the RAM, and executes it. In the case of this embodiment, the CPU analyzes the defocus map of the captured image, determines the degree of blurring of the background based on the defocus amount, and adjusts the sound volume in consideration of the aperture value, F-number, and focal length. Gain calculation processing and the like for adjustment are executed. The details of the processing according to this embodiment realized by the CPU of the central control unit 101 executing the control program will be described later.

The operation unit 111 includes operation devices such as various buttons, switches, and a touch panel provided for the user to operate the imaging apparatus 100, and an operation signal acquisition unit that acquires user operations on these operation devices as operation signals. . Although illustration is omitted, the operation devices of the operation unit 111 include a power button, a shutter button, a video shooting button, and the like.

The storage unit 112 stores various data such as image data obtained by shooting and audio data picked up by the sound pickup unit 108 and the external sound pickup device 200 at the same time as the shooting in association with each other. The storage unit 112 includes a storage medium, and the storage medium includes an internal storage medium (internal memory) of the imaging apparatus and a storage medium such as an SD card that is removable from the imaging apparatus.

A display unit 113 includes a display device such as an LCD (liquid crystal display) or an organic EL display, and displays images as necessary based on image data output from the image processing unit 106 . The display unit 113 also displays an image for a UI (user interface) screen generated by the central control unit 101 . When the operation unit 111 includes a touch panel, the touch panel is provided side by side with the screen of the display unit 113 .

The input/output terminal unit 114 inputs and outputs communication signals and image signals with an external device.
The audio reproduction unit 110 includes a speaker, converts audio data into an electrical signal (analog audio signal), and drives the speaker with the analog audio signal to reproduce audio.
The power supply unit 115 supplies power to each unit (each component) of the image capturing apparatus according to its application.
The power supply control unit 116 performs control to individually start up or shut down the different types of power supplies of the power supply unit 115 .

Hereinafter, taking the case where the external sound collecting device 200 is a wireless microphone as an example, the audio processing of the audio data acquired by the external sound collecting device 200 and the audio data acquired by the sound collecting unit 108, which is a built-in microphone, Description will be made with reference to FIGS. 2 to 4. FIG. In this embodiment, as described above, the external sound pickup device 200 is arranged near the subject and picks up the sound of the subject, and the sound pickup unit 108 picks up the surrounding sounds. In this embodiment, the external sound pickup device 200 is worn by (or held by) a person as a subject, and mainly picks up the person's speaking voice, and the sound pickup unit 108 mainly picks up an image. A description will be given assuming a case of picking up environmental sound, which is the sound of the entire surroundings of the device 100 .

FIG. 2 is a control flowchart of the imaging device 100 according to this embodiment. The processing of each step shown in the flowchart of FIG. 2 is processing executed by the CPU of the central control unit 101, but in the following description, it is assumed that the central control unit 101 executes the processing for simplification of description.

First, when the user presses a movie shooting button that is part of the operation unit 111 of the imaging device 100, the imaging device 100 starts shooting a movie. When moving image shooting is started, an image signal is captured by the imaging unit 104 and processed by the imaging processing unit 105 and is input to the image processing unit 106 .

Also, when moving image shooting is started, the central control unit 101 reads various initial setting values from the ROM as the process of step S201. The initial settings include the gain initially set for the sound picked up by the built-in microphone, which is the sound pickup unit 108, and the initial setting for the sound picked up by the wireless microphone, which is the external sound pickup device 200. Contains gain information. The audio data acquired by the sound pickup unit 108 is input to the audio processing unit 107 . In addition, the reception data transmitted from the external sound collecting device 200 by wireless communication and received by the radio unit 109 is converted into audio data that can be processed by the audio processing unit 107 by the central control unit 101 as described above, and then processed by the audio processing unit 107. It is input to the audio processing unit 107 .

Next, in step S202, the central control unit 101 controls the image processing unit 106 to perform subject detection processing for detecting the main subject from the captured image. In the case of this embodiment, in the subject detection process, the area of the main subject, which is the person, is detected from the captured image of the moving image. That is, the subject detection processing includes processing for separating the main subject and background regions from the image.
Furthermore, in step S203, the central control unit 101 controls the image processing unit 106 to perform image analysis processing, thereby calculating the defocus amount of the background area other than the main subject in the image. That is, the central control unit 101 acquires the defocus amount of the background as information representing the degree of blurring of the background.

Next, in step S<b>204 , the central control unit 101 acquires the aperture value, F-number, focal length, and the like of the photographing optical system unit 102 . The aperture value, F-number, and focal length are information generally obtainable by the imaging device, and details thereof are omitted. In this embodiment, the aperture value, F-number, and focal length are collectively referred to as correction information of the imaging device. Further, the central control unit 101 controls the actuator control unit 103 to move the plurality of optical lenses of the imaging optical system unit 102 in the optical axis direction, thereby performing control for focusing on the main subject. Furthermore, the central control unit 101 may calculate the distance (main subject distance) from the main subject to the imaging device 100 from the position of the optical lens in the optical axis direction when focusing on the main subject.

Next, in step S205, the central control unit 101 controls the sound pickup unit 108 and the external sound pickup device 200 based on the background defocus amount acquired in step S203 and the correction information of the imaging device acquired in step S204. Determines the gain for audio for each microphone. In the case of the present embodiment, as for the gain for the sound from the external sound collection device 200, the initially set gain is used as it is (the volume is not changed). On the other hand, the gain for the sound from the sound pickup unit 108 is determined by adding the correction information of the imaging device to the defocus amount of the background. In the case of this embodiment, the central control unit 101 increases the gain for the sound acquired by the sound pickup unit 108 relative to the gain for the sound acquired by the external sound pickup device 200 as the defocus amount of the background increases. increase in size.

Then, in step S206, the central control unit 101 associates the captured image data with the audio data whose volume has been adjusted by the audio processing unit 107 according to the gain determined in step S205, and stores them in the storage unit 112. Execute the process to be stored in the

Next, in step S<b>207 , the central control unit 101 executes processing for determining whether or not the user has input an instruction to stop movie shooting by pressing the movie shooting button again. In step S207, when the central control unit 101 determines that an instruction to stop moving image shooting has been input, the process of the flowchart in FIG. 2, that is, the moving image recording process ends. On the other hand, if the moving image shooting button has not been pressed again, that is, if the user has not input an instruction to stop shooting the moving image, the central control unit 101 executes the processing from step S208 onward.

In step S208, the central control unit 101 determines whether or not the previous defocus amount acquired in step S203 has changed. When the central control unit 101 determines in step S208 that there is no change in the defocus amount, the process returns to step S202, and the processes after step S202 are executed. On the other hand, when the central control unit 101 determines that the defocus amount has changed in step S208, the process of step S209 is executed.

In step S209, the central control unit 101 determines whether or not the previous correction information of the imaging apparatus acquired in step S204 has changed. When the central control unit 101 determines in step S209 that there is no change in the correction information of the imaging apparatus, the process returns to step S202, and the processes after step S202 are executed. On the other hand, when the central control unit 101 determines in step S209 that there is a change in the correction information of the imaging device, the process returns to step S207, and the processes after step S207 are executed.

FIG. 3 is a diagram showing an example of an imaging situation using the imaging apparatus 100 according to this embodiment. FIG. 3 shows a state in which the imaging apparatus 100 is shooting with a composition in which the main subject 303 and the background subject 304 are within the angle of view. In the example of FIG. 3, the background subject is a house, but the background in this embodiment includes not only the house but also various subjects such as distant buildings and mountains. Also, in the example of FIG. 3, the built-in microphone of the imaging apparatus 100 is represented as a microphone 301 for convenience. On the other hand, an external microphone 302 as an external sound pickup device 200 is arranged near the main subject 303 .

FIG. 4 is a diagram showing an example of how the main subject 403 and the background subject 404 appear in the photographed image when the photographing apparatus according to the present embodiment photographs with the composition shown in FIG. FIG. 4A shows an example in which the background subject 404 blurs less than the main subject 403, and FIG. 4B shows an example in which the background subject 404 blurs more than the main subject 403. there is That is, when photographing is performed as in the composition example shown in FIG. 3, the main subject 403 and the background subject 404 are clearly photographed without blurring as shown in FIG. In some cases, the background subject 404 is blurred with respect to the main subject 403 . For example, when the aperture value is large, the F-number is large, or the focal length is short, the main subject 403 and the background subject 404 are often photographed without being blurred, as shown in FIG. 4A. On the other hand, for example, when the aperture value is small, the F number is small, or the focal length is long, the background subject 404 is often blurred with respect to the main subject 403 as shown in FIG. 4B. Therefore, the central control unit 101 determines the degree of blurring of the background by the defocus amount based on the analysis results such as the defocus map of the captured image. Determine gain.

In the case of this embodiment, the central control unit 101 increases the gain of the environmental sound acquired by the built-in microphone with respect to the gain of the sound of the main subject acquired by the external microphone as the defocus amount of the background increases. Make it relatively large. As a result, the volume of the environmental sound acquired by the built-in microphone increases relative to the volume of the main subject acquired by the external microphone as the degree of blurring of the background increases. That is, when the main subject 301 is near the imaging apparatus 100, the imaging apparatus 100 increases the sound picked up by the sound pickup unit 108 and records the sounds around the main subject 301 relatively. be recorded in large numbers. In other words, the smaller the defocus amount of the background, the more the central control unit 101 adjusts the gain of the environmental sound acquired by the built-in microphone relative to the gain of the sound of the main subject acquired by the external microphone. make it smaller. As a result, the smaller the degree of blurring of the background, the smaller the volume of the environmental sound acquired by the built-in microphone relative to the volume of the main subject acquired by the external microphone. That is, when the main subject 301 is far away from the imaging apparatus 100, the imaging apparatus 100 records the sound picked up by the sound pickup unit 108 at a low volume, so that the surrounding sounds of the main subject 301 are recorded relatively. so that is recorded small.
As a result, the imaging device 100 can record natural environmental sounds according to the distance to the subject.

Note that when the defocus amount of the background is greater than a predetermined threshold, the central control unit 101 sets the gain for the sound picked up by the built-in microphone relative to the gain for the sound picked up by the external microphone near the main subject. can be as large as In other words, when the defocus amount of the background is equal to or less than a predetermined threshold, the central control unit 101 sets the gain for the sound picked up by the built-in microphone to the gain for the sound picked up by the external microphone near the main subject. It may be relatively smaller.

<Second embodiment>
Next, the imaging device 100 of the second embodiment will be described. The configuration of the image pickup apparatus 100 of the second embodiment is the same as that of FIG. 1 described above, and the control flowchart in the image pickup apparatus 100 of the second embodiment is the same as that of FIG. are omitted.

FIG. 5 is a diagram showing an example of an imaging situation by the imaging apparatus 100 of the second embodiment, in which imaging is performed by the imaging apparatus 100 with a composition in which a main subject 503 and a background subject 504 are within the angle of view. It represents the state of being In the case of the first embodiment, the imaging device 100 includes only the built-in microphone of the sound pickup unit 108, and the external microphone that is the external sound pickup device 200 is also an example. A plurality of external microphones as the sound device 200 are connected to the imaging device 100 . In the example of FIG. 5, there are an external microphone 501, an external microphone 502, and an external microphone 505 as the plurality of external microphones included in the external sound pickup device 200, and these are connected to the imaging device 100 by wire or wirelessly. Suppose there is Assume that the external microphone 501 is arranged near the imaging apparatus 100, the external microphone 502 is arranged near the main subject 503, and the external microphone 505 is arranged near the background subject 504, respectively. In the case of the second embodiment, among these microphones, the built-in microphone of the imaging device 100, the external microphone 501 near the imaging device 100, and the external microphone 501 near the background subject 504 each pick up ambient environmental sounds. do. That is, in the case of the second embodiment, the built-in microphone of the imaging device 100 corresponds to the first microphone, and the external microphone 501 near the imaging device 100 and the external microphone 501 near the background object 504 correspond to the third microphone. . On the other hand, an external microphone 502 near the main subject 503 picks up the voice of the main subject 503 . That is, in the case of the second embodiment, the external microphone 502 near the main subject 503 corresponds to the second microphone. Then, the imaging device 100 of the second embodiment records audio data picked up by the built-in microphones of the imaging device 100 and the external microphones 501, 502, and 505 together with the captured image.

Here, in the second embodiment, the imaging apparatus 100 performs focus control on the main subject 503 as in the above-described embodiments. Therefore, even if the main subject 503 moves, the main subject 503 does not blur. Suppose it is taken. On the other hand, since the background subject 504 is out of focus, the background subject 504 is blurred. Varies depending on aperture value, f-number, and focal length.

FIGS. 5A and 5B show an example in which the main subject 503 moves within the composition in which the main subject 503 and the background subject 504 are within the angle of view. FIG. 5A shows an example in which the main subject 503 is near the background subject 504 and far from the imaging apparatus 100, and FIG. An example away from the background object 504 is shown. 5A and 5B show the case where the main subject 503 moves from the position shown in FIG. 5A to the position shown in FIG. It is assumed that an example of moving from the position to the position shown in FIG. 5(A) is shown. Here, when the imaging apparatus 100 is performing focus control on the main subject 503, the defocus amount representing the degree of blurring of the background subject 504 is as shown in FIG. becomes larger. 5A and 5B, when the main subject 503 moves in the perspective direction with respect to the image capturing apparatus 100, the background subject 504 changes according to the movement of the main subject 503. The defocus amount increases or decreases.

In the second embodiment, similarly to the above embodiments, the central control unit 101 of the imaging apparatus 100 determines the degree of blurring of the background subject 504 based on the defocus amount of the defocus map of the captured image. Then, the central control unit 101 takes into account the defocus amount of the background object 504 and the correction information of the imaging device, and determines the gain for the sound of each microphone. However, in the second embodiment, as in the first embodiment, the gain of the sound of the external microphone 502 near the main subject 503 is not adjusted. Therefore, based on the defocus amount of the background subject 504, the central control unit 101 adjusts the gain for the sound of the built-in microphone of the image capturing apparatus 100, the external microphone 501 near the image capturing apparatus 100, and the external microphone 505 near the background subject 504. decide. That is, the central control unit 101 sets gains such that the volume of the built-in microphone, the external microphone 501, and the external microphone 505 increases as the defocus amount of the background object 504 increases. On the other hand, the central control unit 101 sets gains such that the volumes of the built-in microphone, the external microphone 501, and the external microphone 505 decrease as the defocus amount of the background object 504 decreases.

Furthermore, in the case of the second embodiment, when the main subject moves in the perspective direction with respect to the imaging apparatus 100, the central control unit 101 adjusts the gain obtained based on the defocus amount of the background subject 504 to the main subject. The gain corresponding to the increase/decrease of the defocus amount due to movement is adjusted. The central control unit 101 controls the audio processing unit 107 so as to adjust the volume of the built-in microphone, the external microphone 501 near the imaging device 100, and the external microphone 505 near the background object 504 based on the gain after the subtraction. .

That is, in the imaging apparatus 100 of the second embodiment, when the main subject moves in the far and near direction with respect to the imaging apparatus, the main subject A gain corresponding to the increase or decrease of the defocus amount due to the movement of the is added or subtracted. As a result, even when the main subject moves in the perspective direction, the imaging apparatus of the second embodiment can adjust the sound in the imaging direction of the imaging apparatus, that is, in the depth direction. It is possible to reproduce sound with feeling.

When the main subject moves in the perspective direction with respect to the imaging device, the gain corresponding to the increase or decrease in the defocus amount due to the movement of the main subject is added or subtracted from the gain obtained based on the defocus amount of the background. The processing to do so can also be applied to the imaging apparatus of the first embodiment described above.

<Third Embodiment>
Next, an imaging device 100 according to a third embodiment will be described. The configuration of the imaging apparatus 100 of the third embodiment is the same as that of FIG. 1 described above, and the control flowchart in the imaging apparatus 100 of the third embodiment is generally the same as that of FIG. 2 described above. Description is omitted.

In the first embodiment described above, the greater the degree of blurring of the background, the greater the gain for the sound picked up by the built-in microphone of the imaging device, etc. relative to the gain for the sound picked up by the external microphone near the main subject. Although the gain is increased, conversely, the gain for the environmental sound may be decreased. In other words, the smaller the defocus amount of the background, the larger the gain of the sound picked up by the built-in microphone relative to the gain of the sound picked up by the external microphone near the subject. Further, when the defocus amount of the background is larger than a predetermined threshold, the gain for the sound picked up by the built-in microphone may be relatively smaller than the gain for the sound picked up by the external microphone near the main subject. .

The imaging apparatus 100 of the third embodiment is an example in which the user can arbitrarily switch and set the audio gain determined based on the defocus amount of the background via the operation unit 111 . The imaging device 100 of the third embodiment makes the gain of the sound picked up by the built-in microphone relatively larger than the gain of the sound picked up by the external sound pickup device 200 according to the defocus amount of the background. It has a function that allows switching between setting and decreasing setting according to a user operation.

<Fourth Embodiment>
In the above-described first to third embodiments, the sound volume of each sound picked up by a plurality of microphones is adjusted according to the degree of blurring of the background at the same time as the image is captured by the imaging device, and the sound data after the adjustment is converted into the captured image. I gave an example of recording with the data of The imaging apparatus 100 of the fourth embodiment records each audio data picked up by a plurality of microphones at the same time as shooting without adjusting the recording volume. In the fourth embodiment, automatic editing such as automatically adjusting the volume by determining the gain for the audio data based on the defocus amount of the background and the correction information of the imaging device when the moving image is edited later. A device with functions will be described. The automatic editing function involving volume adjustment according to the fourth embodiment may be provided by the imaging device 100, or may be realized by an information processing device such as a personal computer, a smartphone, or a tablet terminal by executing an application program. .

In the present embodiment, an example in which the imaging apparatus 100 has an automatic editing function with volume adjustment will be described. Note that the configuration of the imaging apparatus 100 of the fourth embodiment is the same as that of FIG. 1 described above, and therefore the illustration thereof is omitted.

FIG. 6 is a control flowchart in the imaging device 100 of the fourth embodiment. The processing of each step shown in the flowchart of FIG. 6 is processing executed by the CPU of the central control unit 101, and for the sake of simplification, it will be assumed that the processing is executed by the central control unit 101 in the following description. FIG. 7A is a control flow chart for moving image shooting, audio pickup, and recording thereof in the imaging apparatus 100 . FIG. 7B shows automatic editing processing in which a recorded moving image and audio are reproduced during moving image editing, and the gain for the audio is determined based on the defocus amount of the background and the correction information of the imaging device to adjust the volume, and the editing processing. FIG. 10 is a control flow chart until the later audio data is associated with an image and recorded. FIG. Note that the processing of steps S601 to S604 and steps S607 to S609 in FIG. 7A is the same as the processing of steps S201 to S204 and steps S207 to S209, which are the corresponding processing steps in FIG. omitted.

After the process of step S604 in FIG. 7A, in step S605, the central control unit 101 determines the recording volume for each microphone of the sound pickup unit 108 and the external sound pickup device 200. FIG. In the case of the fourth embodiment, the recording volume of each microphone is the volume corresponding to the gain initially set for each microphone.

Next, in step S606, the central control unit 101 converts the sound data picked up by each microphone, the defocus amount acquired in step S603, and the correction information of the imaging device acquired in step S604 into the captured image time. It is stored in the storage unit 112 in association with the code.

Next, the control flowchart of the moving image editing process in FIG. 7B will be described. In this embodiment, an example is given in which the imaging apparatus 100 has an automatic editing function that determines the audio gain based on the background defocus amount and the correction information of the imaging apparatus and adjusts the volume. Note that in the present embodiment, editing processing for moving images is an existing processing, so a detailed description thereof will be omitted, and volume adjustment processing that is performed along with moving image editing will be mainly described.

When the user starts the automatic editing process, the central control unit 101 reads the time code of the captured image from the storage unit 112 in step S611. At the same time, the central control unit 101 reads from the storage unit 112 the audio data of each microphone, the defocus amount, and the correction information of the imaging device recorded corresponding to the time code. The audio data of each microphone is the audio data acquired and recorded by the built-in microphone of the imaging device 100 and the audio data acquired and recorded by the external microphone which is the external sound pickup device 200 .

Next, in step S612, the central control unit 101 adjusts the volume of the audio data of each microphone based on the background defocus amount and the correction information of the imaging device. In the fourth embodiment, the central control unit 101 adjusts the volume of audio data picked up by the built-in microphone with a gain determined based on the defocus amount of the background and the correction information of the imaging device. make adjustments. On the other hand, the central control unit 101 regards the voice data of the external microphone as voice data of the recorded volume. Then, the central control unit 101 determines and synthesizes the sound of the built-in microphone, that is, the sound after adjustment described above and the sound of the external microphone at each timing according to the timing code.

After that, in step S613, the central control unit 101 stores the voice data synthesized in the process of step S612 and the image data in the storage unit 112 as the edited image and voice. After that, the central control unit 101 terminates the automatic editing process.

In the fourth embodiment, an example in which the imaging device adjusts the volume has been described. is executed in The information processing apparatus in this case executes an application program that implements the automatic editing process according to the fourth embodiment. Although illustration of the hardware configuration is omitted, an information processing device that realizes automatic editing processing includes a CPU, a ROM, a RAM, an auxiliary storage device, a display unit, an operation unit, a communication I/F, a bus, and the like. be done. The CPU uses the computer programs and data stored in the ROM and RAM to control the entire apparatus and to perform the automatic editing process including the volume adjustment described above. Further, the information processing apparatus may have one or a plurality of pieces of dedicated hardware different from the CPU, and may be configured such that at least part of the processing by the CPU is executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM stores programs and the like that do not require modification. The RAM temporarily stores programs and data supplied from the auxiliary storage device, data supplied from the outside via the communication I/F, and the like. The auxiliary storage device is composed of an HDD, SSD, or the like, and stores various data such as image data, audio data, time code, defocus amount, auxiliary information of the imaging device, and other control parameters. The display unit is composed of, for example, a liquid crystal display, an LED display, or the like, and displays a GUI (graphical user interface) or the like for the user to operate the information processing apparatus. The operation unit includes, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU in response to user's operations. The CPU also operates as a display control section that controls the display section and as an operation control section that controls the operation section. The communication I/F is used for communication with an external device of the information processing device. For example, when the information processing device is further connected to an external device by wire, a communication cable is connected to the communication I/F. If the information processing device has a function of wirelessly communicating with an external device, the communication I/F has an antenna. The bus connects each part of the information processing device to transmit information. Note that in the case of the fourth embodiment, the external device connected to the information processing device is the imaging device described above, another information processing device, or the like. Further, although the display section and the operation section are assumed to exist inside the information processing apparatus, at least one of the display section and the operation section may exist as a separate device outside the information processing apparatus. Further, the information processing device does not necessarily have to include a display unit and an operation unit.

FIG. 7 is a diagram used for explaining an example of a subject and an imaging device assuming the fourth embodiment. In FIG. 7, it is assumed that the external microphone 705 is arranged near the main subject 701 on which the imaging apparatus 100 is focused. Also, in the example of FIG. 7, subjects 702 and 703 other than the main subject 701 are present. However, since the imaging apparatus 100 focuses on the main subject 701, the other subjects 702 and 703 are blurred to some extent. When the imaging apparatus 100 shoots a moving image, not only the sound picked up by the external microphone 705 near the main subject 701 but also the sounds of the other subjects 702 and 703 are included in the environmental sound. Recorded.

After that, when automatic editing processing is executed by the imaging device 100, the voices of the subjects 702 and 703 that are out of focus by the imaging device 100 are The volume is relatively adjusted according to the degree of blurring.

As described above, according to the imaging apparatuses of the first to fourth embodiments, it is possible to appropriately adjust the sound in the shooting direction, that is, in the depth direction, and to provide a sense of realism with a clear sense of perspective for the shot image. It becomes possible to play a certain sound.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.
All of the above-described embodiments merely show specific examples for carrying out the present invention, and the technical scope of the present invention should not be construed to be limited by these. That is, the present invention can be embodied in various forms without departing from its technical concept or main features.

100: imaging device, 101: central control unit, 102: optical system unit, 104: imaging unit, 106: image processing unit, 107: audio processing unit, 108: sound pickup unit, 109: radio unit, 110: audio reproduction unit , 112: storage unit, 113: display unit, 200: external sound pickup device

Claims (15)

imaging means;
a first microphone;
connecting means for connecting a second microphone;
adjusting means for adjusting a gain for the audio signal input from the first microphone based on the defocus amount of the image captured by the imaging means;
An imaging device characterized by comprising: The first microphone is a built-in microphone of the imaging device,
2. The imaging apparatus according to claim 1, wherein said second microphone is a microphone wirelessly connected via said connection means. The adjustment means obtains a defocus amount of a background area excluding an area of a subject to be photographed in the captured image, and adjusts the gain based on the defocus amount of the background area. 3. The imaging device according to claim 1 or 2, characterized by: The adjusting means adjusts the gain of the audio signal input from the first microphone relative to the gain of the audio signal input from the second microphone as the defocus amount of the background area increases. 4. The image pickup apparatus according to claim 3, wherein the image pickup apparatus according to claim 3, wherein the image pickup apparatus adjusts to a large extent. When the defocus amount of the background region is greater than a threshold, the adjustment means adjusts the gain for the audio signal input from the first microphone to the gain for the audio signal input from the second microphone. 4. The imaging apparatus according to claim 3, wherein the adjustment is made relatively large with respect to. The adjusting means adjusts the gain of the audio signal input from the first microphone relative to the gain of the audio signal input from the second microphone as the defocus amount of the background area increases. 4. The image pickup apparatus according to claim 3, wherein the image pickup apparatus is adjusted to be relatively small. When the defocus amount of the background region is greater than a threshold, the adjustment means adjusts the gain for the audio signal input from the first microphone to the gain for the audio signal input from the second microphone. 4. The image pickup apparatus according to claim 3, wherein the adjustment is made relatively small. The adjustment means adjusts the gain for the audio signal input from the first microphone relative to the gain for the audio signal input from the second microphone, according to the defocus amount of the background area. 4. The setting according to claim 3, characterized in that the setting for adjusting the gain to the audio signal input from the second microphone is switched according to the user's operation between a setting for adjusting the gain relatively large and a setting for adjusting the gain of the audio signal input from the second microphone relatively small. The imaging device described. When the object to be photographed moves, the adjustment means corresponds to the amount of increase or decrease in the defocus amount of the background area due to the movement of the object, with respect to the gain adjusted based on the defocus amount of the background area. 9. The imaging apparatus according to any one of claims 3 to 8, wherein the gain is adjusted. A third microphone is also connected to the connection means,
10. The adjustment device according to any one of claims 1 to 9, wherein the adjusting means also adjusts the gain of the audio signal input from the third microphone based on the defocus amount of the captured image. 10. The image pickup device according to claim 1. The third microphone is arranged in the vicinity of the imaging device and is connected to the connection means by wire or wirelessly, and is arranged at a position different from the position of the subject to be photographed and is connected by wire or wirelessly to the connection means. 11. Imaging device according to claim 10, comprising at least one of a microphone connected to means. 3. The adjustment means adjusts the gain by taking into account at least one of an aperture value, an F value, and a focal length in an optical system that forms an optical image on the imaging means. 12. The imaging device according to any one of 11. 13. The apparatus according to any one of claims 1 to 12, further comprising recording means for recording the audio signal after gain adjustment, the audio signal input from the second microphone, and the captured image. The imaging device according to any one of items 1 and 2. A control method for an imaging device having imaging means, a first microphone, and a connection means for connecting a second microphone, comprising:
A control method for an imaging device, comprising a control step of adjusting a gain for an audio signal input from said first microphone based on a defocus amount of an image captured by said imaging means. A program for causing a computer of an imaging device to function as adjustment means for the imaging device according to any one of claims 1 to 13.

Images