JP2018207313A - Audio processing device and method of controlling the same - Google Patents

Abstract

To obtain a noise-reduced stereo audio signal with two-microphone constitution for stereo sound collection without adding any extra microphone.SOLUTION: An audio processing device has: a first microphone of a stereo microphone which is housed in the device at a predetermined position, and converts an audio entering from a first opening part provided at a predetermined position as a main target into an electric signal; and a second microphone which is housed in the device at a position where the microphone functions as the other microphone, and converts an audio entering from a second opening part, having smaller area than the first opening part, into an electric signal. Since driving noise from a predetermined driving part that the audio processing device has is propagated to the second microphone, the capacity of a space between the second microphone and second opening part is larger than the capacity between the first microphone and first opening part. The audio processing device uses and reversely converts audio signal from the first and second microphones into noise-reduced audio signals, and outputs the audio signals.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an audio processing technique in an apparatus having a drive mechanism.

  An imaging device represented by a digital camera or a video camera can capture a moving subject, record the moving image data obtained as a result, and also record the sound around the subject. Hereinafter, the sound around the subject to be recorded is hereinafter referred to as “ambient environmental sound”.

  Further, the imaging apparatus can focus or zoom the moving subject during imaging by moving the optical lens. Here, the movement of the optical lens is mechanically performed, and a driving sound is generated when the lens is moved. If this driving sound is superimposed on the ambient environmental sound, the quality of the moving image with sound is impaired. For this reason, it has been conventionally desired to reduce such driving noise.

JP 2011-114465 A

  Patent Document 1 is known as a document disclosing a method for reducing drive noise generated from a drive unit or the like of an apparatus. In this patent document 1, a noise detection microphone is mounted in an imaging apparatus in addition to a normal audio signal acquisition microphone. According to Patent Document 1, the imaging apparatus includes a first microphone for recording sound outside the apparatus and a second microphone for recording noise generated inside the apparatus. In addition, the first microphone faces the outside of the device and the second microphone faces the inside of the device, thereby detecting noise generated inside the device. However, the technique disclosed in Patent Document 1 is to add a microphone for detecting noise in addition to a microphone necessary for an audio channel to be recorded. That is, since a microphone other than the microphone intended for recording is mounted, a problem of cost and area occurs. In addition, Patent Document 1 is configured to record monaural audio. For example, in order to perform stereo audio recording, two microphones for stereo and one for noise are required. .

  The present invention has been made in view of such a problem, and it is possible to perform stereo sound collection and noise detection without adding a new microphone to a configuration including two microphones for performing stereo sound recording. It is intended to provide possible technology.

In order to solve this problem, for example, the speech processing apparatus of the present invention has the following configuration. That is,
A housing,
A drive unit;
A first microphone housed in the housing such that sound is propagated through a first opening provided at a first predetermined position of the housing;
The sound is propagated through a second opening having a smaller area than the first opening provided at a second predetermined position of the housing related to the first predetermined position. 2, wherein the volume of the second space between the second microphone and the second opening is the first space between the first microphone and the first opening. The second microphone housed inside the housing to be larger than the volume of
Conversion means for converting time-series audio data obtained from the first microphone into first frequency spectrum data and time-series audio data obtained from the second microphone into second frequency spectrum data; ,
Calculating means for calculating the amount of noise by the driving unit for each frequency from the first frequency spectrum data and the second frequency spectrum data obtained by the converting means;
Based on the first frequency spectrum data, the second frequency spectrum data, and the amount of the noise calculated by the calculation means, the frequency spectrum data of the left channel in which the noise is suppressed, and the right channel Generating means for generating frequency spectrum data;
Inverse conversion means for inversely converting the frequency spectrum data of the left and right channels generated by the generation means into audio data of the time-series left and right channels.

  According to the present invention, it is possible to obtain a stereo audio signal with reduced noise without adding a new microphone, with the configuration of two microphones for collecting stereo sound.

1 is a block configuration diagram of an imaging apparatus according to an embodiment. The detailed block block diagram of the imaging part of the imaging device of embodiment, and an audio | voice input part. The mechanical block diagram of the audio | voice input part of the imaging device of embodiment. 6 is a flowchart illustrating a REC sequence of the imaging apparatus according to the embodiment. 4 is a timing chart of an L / Rch generation unit of the imaging apparatus according to the embodiment. FIG. 2 is a block diagram illustrating a detailed configuration of a voice input unit of the imaging apparatus according to the embodiment. The figure which shows the system which propagates ambient environment sound to the imaging device. The figure which shows the relationship of the phase of the frequency spectrum from the main microphone a of the imaging device of embodiment, and the frequency spectrum from the submicrophone b. The figure which shows the relationship between the emphasis coefficient of the stereo feeling of embodiment, and a frequency. The figure which shows the amplitude spectrum for each frequency of each of the main microphone a and the sub microphone b of the imaging apparatus of the embodiment. The figure which shows the time-series amplitude spectrum of the frequency N point of the sub microphone b of the imaging device of embodiment. It is a figure which shows the phase of a time series of the main microphone a and the submicrophone b of the imaging device of embodiment. 6 is an operation timing chart of an Mch-Sch operation unit of the imaging apparatus according to the embodiment. 6 is an operation timing chart of a sensitivity difference correction unit of the imaging apparatus according to the embodiment. The mechanical block diagram of the audio | voice input part of the imaging device of embodiment. The figure which shows the frequency spectrum from the main microphone a of the imaging device of embodiment, and the frequency spectrum from the submicrophone b. The figure which shows the frequency relationship of the wind noise gain with respect to the wind noise level of embodiment. The figure which shows the relationship between the ratio and frequency with which the frequency spectrum from the main microphone a of the imaging device of embodiment and the frequency spectrum from the sub microphone b are synthesize | combined. The timing chart which changes the emphasis coefficient used for emphasizing a stereo effect according to the time of driving noise detection and the time of wind noise detection about the stereo suppression part of the imaging device of an embodiment. The figure which shows the relationship of the emphasis coefficient used for emphasis of a synthetic | combination ratio, frequency, and a stereo effect at the time of the wind noise detection of embodiment. The figure which shows the time constant of the drive noise removal gain of the embodiment, the wind noise subtraction amount, the stereo gain for Lch generation, and the stereo gain for Rch generation.

  Embodiments according to the present invention will be described below in detail with reference to the drawings. In the present embodiment, a voice processing device accommodated in an imaging device will be described.

  FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus 100 according to the embodiment. The imaging apparatus 100 includes an imaging unit 101, a voice input unit 102, a memory 103, a display control unit 104, and a display unit 105. In addition, the imaging apparatus 100 includes an encoding processing unit 106, a recording / playback unit 107, a recording medium 108, a control unit 109, an operation unit 110, an audio output unit 111, a speaker 112, an external output unit 113, and a bus connecting them. 114.

  The imaging unit 101 converts an optical image of a subject captured by a photographing optical lens into an image signal by an imaging element, performs analog-digital conversion, image adjustment processing, and the like, and generates image data. The photographing optical lens may be a built-in optical lens or a detachable optical lens. Further, the imaging element may be a photoelectric conversion element typified by a CCD, a CMOS or the like.

  The audio input unit 102 collects peripheral audio from outside the audio processing device (in the embodiment, outside the imaging device) with a built-in microphone or connected via an audio terminal, and generates an electrical signal. The audio input unit 102 performs analog-digital conversion, audio processing, and the like to generate audio data. The microphone may be directivity or omnidirectional, but in this embodiment, an omnidirectional microphone is used.

  The memory 103 is used for temporarily storing image data obtained by the imaging unit 101 and audio data obtained by the audio input unit 102.

  The display control unit 104 displays an image related to the image data obtained by the imaging unit 101, an operation screen of the imaging apparatus 100, a menu screen, and the like on the display unit 105 or an external display via a video terminal (not shown). . Although the kind of the display part 105 is not ask | required, it is a liquid crystal display, for example.

  The encoding processing unit 106 reads out image data and audio data temporarily stored in the memory 103, performs predetermined encoding, and generates compressed image data, compressed audio data, and the like. Further, the audio data may not be compressed. The compressed image data is, for example, MPEG2 or H.264. It may be compressed by any compression method such as H.264 / MPEG4-AVC. The compressed audio data may also be compressed by a compression method such as AC3 (A) AC, ATRAC, ADPCM. The encoding processing unit 106 also performs decoding processing of the encoded data (compressed image data and compressed audio data).

  The recording / reproducing unit 107 records the compressed image data, compressed audio data or audio data, and various data generated by the encoding processing unit 106 on the recording medium 108, and reads out from the recording medium 108. Here, the recording medium 108 is a non-volatile recording medium that records image data, audio data, and the like. For example, there are a magnetic disk, an optical disk, a semiconductor memory, and the like. Further, the recording medium 108 may be fixed to the apparatus 100 or detachable.

  The control unit 109 controls each block of the image capturing apparatus 100 by transmitting a control signal to each block of the image capturing apparatus 100 via the bus 114. From the CPU, memory, or the like for executing various controls. Composed. The memory used by the control unit 109 is a ROM that stores various control programs, a RAM that is used as a work area for arithmetic processing, and the like, and includes an external memory of the control unit 109.

  The operation unit 110 is a button, a dial, a touch panel, or a combination thereof, and transmits an instruction signal to the control unit 109 according to a user operation. Specifically, the operation unit 110 includes a shooting button for instructing start and end of moving image recording, a zoom lever for instructing to perform an optical or electronic zoom operation on the image, and a cross for performing various adjustments. Key, enter key, etc.

  The audio output unit 111 outputs the audio data and compressed audio data reproduced by the recording / reproducing unit 107 or the audio data output by the control unit 109 to the speaker 112 and the audio terminal. The external output unit 113 outputs the compressed video data, compressed audio data, audio data, and the like reproduced by the recording / reproducing unit 107 to an external device. The data bus 114 supplies various data such as audio data and image data and various control signals to each block of the imaging apparatus 100.

  The above is the description of the configuration of the imaging device 100 according to the embodiment. Next, a normal operation of the imaging apparatus in the embodiment will be described.

  In the imaging apparatus 100 according to the present embodiment, power from a power supply unit (not illustrated) is supplied to each block of the imaging apparatus in response to a user operating the operation unit 110 and giving an instruction to turn on the power. Is done.

  When the power is supplied, the control unit 109 confirms which mode, for example, the shooting mode or the reproduction mode is designated by the mode change switch of the operation unit 110 by an instruction signal from the operation unit 110. In the moving image recording mode in the shooting mode, the image data obtained by the imaging unit 101 and the audio data obtained by the audio input unit 102 are stored as one image file. In the reproduction mode, the image file recorded on the recording medium 108 is reproduced by the recording / reproducing unit 107, displayed on the display unit 105, and output from the speaker 112.

  In the shooting mode, first, the control unit 109 transmits a control signal to each block of the imaging apparatus 100 so as to shift to the shooting standby state, and performs the following operation.

  The imaging unit 101 converts an optical image of a subject captured by a photographic optical lens into a moving image signal using an imaging element, performs analog-digital conversion, image adjustment processing, and the like to generate moving image data. Then, the imaging unit 101 transmits the obtained moving image data to the display processing unit 104 and causes the display unit 105 to display the moving image data. The imaging unit 101 outputs a moving image signal in which one frame is horizontal 1920 pixels × vertical 1080 pixels and the frame rate is 30 frames / second. The user prepares for shooting while viewing the screen displayed in this way.

  The audio input unit 102 converts analog audio signals obtained by a plurality of microphones into digital signals, processes the obtained digital audio signals, and generates multi-channel audio data. Then, the obtained audio data is transmitted to the audio output unit 111 and is output as audio from the connected speaker 112 or an unillustrated earphone. The user can also adjust the manual volume to determine the recording volume while listening to the sound output in this way.

  Next, when a shooting start instruction signal is transmitted to the control unit 109 by the user operating the recording button of the operation unit 110, the control unit 109 transmits a shooting start instruction signal to each block of the imaging apparatus 100. Then, the moving image recording mode in the shooting mode is entered. Specific processing of the control unit 109 is as follows.

  The imaging unit 101 converts an optical image of a subject captured by a photographic optical lens into a moving image signal using an imaging element, performs analog-digital conversion, image adjustment processing, and the like to generate moving image data. Then, the obtained moving image data is transmitted to the display processing unit 104 and displayed on the display unit 105. Further, the imaging unit 101 transmits the obtained image data to the memory 103.

  The audio input unit 102 converts analog audio signals obtained by a plurality of microphones into digital signals, processes the obtained digital audio signals, and generates multi-channel audio data. Then, the obtained audio data is transmitted to the memory 103. If there is only one microphone, the obtained analog audio signal is digitally converted to generate audio data, and the audio data is transmitted to the memory 103.

  The encoding processing unit 106 reads moving image data and audio data temporarily stored in the memory 103 and performs predetermined encoding, generates compressed moving image data, compressed audio data, and the like, and stores them in the memory 103 again. To do.

  The control unit 109 combines the compressed moving image data and the compressed audio data stored in the memory 103 to form a data stream, and outputs the data stream to the recording / reproducing unit 107. When the audio data is not compressed, the control unit 109 combines the audio data stored in the memory 103 and the compressed moving image data, forms a data stream, and outputs the data stream to the recording / reproducing unit 107.

  The recording / playback unit 107 writes the data stream to the recording medium 108 as one moving image file under the management of a file system such as UDF or FAT.

  The imaging apparatus 100 continues the above process during the moving image recording state. When the user operates the recording button of the operation unit 110 and a shooting end instruction signal is transmitted to the control unit 109, the control unit 109 transmits a shooting end instruction signal to each block of the imaging apparatus 100. The following operations are performed.

  The imaging unit 101 and the audio input unit 102 stop generating moving image data and audio data, respectively. The encoding processing unit 106 reads the remaining image data and audio data stored in the memory, performs predetermined encoding, and stops operation when generation of compressed moving image data, compressed audio data, and the like is completed. When the audio data is not compressed, the operation is naturally stopped when the generation of the compressed moving image data is finished.

  Then, the control unit 109 synthesizes the last compressed moving image data and the compressed audio data or audio data, forms a data stream, and outputs the data stream to the recording / reproducing unit 107.

  The recording / playback unit 107 writes the data stream to the recording medium 108 as one moving image file under the management of a file system such as UDF or FAT. When the supply of the data stream is stopped, the moving image file is completed and the recording operation is stopped.

  When the recording operation stops, the control unit 109 transmits a control signal to each block of the imaging apparatus 100 so as to shift to the shooting standby state, and returns to the shooting standby state.

  Next, the playback mode will be described. When the user operates the operation unit 110 to enter the reproduction mode, the control unit 109 transmits a control signal to each block of the imaging apparatus 100 so as to shift to the reproduction state, and performs the following operation.

  The recording / playback unit 107 reads a moving image file composed of compressed moving image data and compressed audio data recorded on the recording medium 108, and sends the read compressed moving image data and compressed audio data to the encoding processing unit 106.

  The encoding processing unit 106 decodes the compressed moving image data and the compressed audio data, and transmits them to the display control unit 104 and the audio output unit 111, respectively. The display control unit 104 causes the display unit 105 to display the decoded moving image data. The audio output unit 111 outputs the decoded audio data to the built-in speaker 112 or an attached external speaker and reproduces it as sound.

  As described above, the imaging apparatus 100 of the present embodiment can perform recording and reproduction of moving images and sounds.

  In the present embodiment, when obtaining an audio signal, the audio input unit 102 performs processing such as level adjustment processing of the audio signal obtained by the microphone. This process may always be performed after the apparatus is activated, or may be performed after the photographing mode is selected. Alternatively, it may be performed after a mode related to audio recording is selected. Further, in a mode related to audio recording, the above processing may be performed in response to the start of audio recording. In the present embodiment, description will be made assuming that the above processing is performed at the timing when moving image shooting is started.

  FIG. 2 is a block configuration diagram of the imaging unit 101 and the audio input unit 102 of the imaging apparatus 100 of the present embodiment.

  The imaging unit 101 includes an optical lens 201 that captures an optical image of a subject, and an imaging element 202 that converts the optical image of the subject captured by the optical lens 201 into an electrical signal (image signal). Furthermore, the imaging unit 101 includes an image processing unit 203 that converts an analog image signal obtained by the imaging element 202 into a digital image signal, performs image quality adjustment processing, forms image data, and transmits the image data to a memory. . Furthermore, the imaging unit 101 includes an optical lens control unit 204 having a known drive mechanism such as a position sensor for moving the optical lens 201, a motor, and the like. In the present embodiment, it is described that the image pickup unit 101 includes the optical lens 201 and the optical lens control unit 204. However, the optical lens 201 is an interchangeable lens that is detachable from the image pickup apparatus 100 via a lens mount. It may be. Further, the optical lens control unit 204 may be provided in the interchangeable lens.

  Here, when a user operates the operation unit 110 to input instructions such as zoom operation and focus adjustment, the control unit 109 causes the optical lens control unit 204 to move the optical lens 201 (control signal (drive signal)). Send. In response to this control signal, the optical lens control unit 204 confirms the position of the optical lens 201 with a position sensor (not shown), and moves the optical lens 201 with a motor (not shown). In addition, when the control unit 109 confirms the image obtained by the image processing unit 203 and the distance to the subject and automatically adjusts the distance, a control signal for driving the optical lens is transmitted. In addition, when a so-called image stabilization function for preventing image blurring is provided, the control unit 109 controls the optical lens 201 based on vibration detected by a vibration sensor (not shown). Is transmitted to the optical lens control unit 204.

  At this time, driving noise due to movement of the optical lens 201 and driving noise of a motor for moving the optical lens 201 are generated. In response to a control signal for driving the optical lens 201 from the control unit 109, the optical lens control unit 204 drives the optical lens 201. Therefore, the control unit 109 can know (detect or determine) the timing at which driving noise occurs.

  In the present embodiment, under the control of the optical lens 201, for example, zooming at a maximum of 50 times and at a minimum of 1 time can be optically performed. This is called optical zoom in this embodiment. Of course, the magnification of the optical zoom may be above or below. In the optical zoom, the optical lens control unit 204 moves the optical lens of the optical lens 201 in accordance with an instruction from the control unit 109 to zoom the optical image of the subject. The image processing unit 203 also has an electronic zoom function that outputs an image signal obtained by zooming in on a part of the image signal obtained by the image sensor 202. Also, an electronic zoom function is provided that outputs an image signal obtained by widening the range of an image obtained by the image sensor 202 and zooming out the image size by the image processing unit 203.

  The above is the configuration and operation of the imaging unit 101 in the embodiment. Next, the configuration and operation of the voice input unit 102 will be described.

  The imaging apparatus 100 according to the embodiment includes two microphones denoted by reference numerals 205a and 205b. These microphones 205a and 205b convert vibrations propagating through the air (medium) into electrical signals and output audio signals. The microphone 205a is a main (MAIN) microphone, and the microphone 205b is a sub (SUB) microphone 205b.

  As will be described in detail later, the main microphone 205a functions as a microphone corresponding to one channel of stereo sound and mainly acquires sound from outside the sound processing apparatus (in the embodiment, outside the imaging apparatus 100). It is a microphone for doing. The sub microphone 205b is arranged at a position that functions as a microphone corresponding to the other channel of stereo sound. The sub microphone 205b is a microphone for mainly acquiring driving noise from the driving unit in the sound processing apparatus (the imaging apparatus 100) as compared with the main microphone 205a.

  The main microphone 205a outputs an analog audio signal as Mch (main channel), and the sub microphone 205b outputs an analog audio signal as Sch (subchannel). In the present embodiment, the first sound input unit is the main microphone 205a, and the first sound signal is Mch. The second audio input unit is the sub microphone 205b, and the second audio signal is Sch. In this embodiment, since the stereo system is configured with two channels, the arrangement positions of the main microphone 205a and the sub microphone 205b are provided at positions separated by a predetermined distance in the horizontal direction when the imaging unit 101 is held upright. Yes. In the embodiment, the number of microphones is 2. However, a configuration in which more microphones are held may be used.

  The analog audio signals obtained by the main microphone 205a and the sub microphone 205b are supplied to the A / D conversion unit 206, where each audio signal is converted into digital audio data. The A / D conversion unit 206 in this embodiment performs sampling at a sampling rate of 48 KHz and generates 16-bit digital data per sampling.

  The time-sequential digital audio data obtained by the A / D conversion unit 206 in a predetermined period (frame) of the audio signal is supplied to the FFT unit 207, where it is subjected to fast Fourier transform, and a frequency spectrum for each frequency. Converted to data. In this embodiment, the frequency spectrum is converted as 1024 point frequency spectrum data from 0 Hz to 48 kHz, and has a 512 point frequency spectrum up to 24 kHz which is the Nyquist frequency. The frequency spectrum data from the main microphone 205a is represented as Main [0] to [511], and the frequency spectrum data from the sub microphone 205b is represented as Sub [0] to [511]. In the present embodiment, the first sound spectrum data is represented as Main [0] to [511], and the second sound spectrum data is represented as Sub [0] to [511]. It is assumed that the subscript “0” of each spectrum data represents the minimum frequency and “511” represents the maximum frequency.

  The drive sound calculation processing unit 209 determines a drive noise subtraction amount for each frequency component of the frequency spectrum data obtained by the FFT unit 207 in accordance with a control signal from the control unit 109 for driving the drive unit. . This driving noise is generated when the optical lens 201 is driven. Note that the drive unit in the present embodiment refers to the optical lens 201 that is driven by zoom operation and focus adjustment. The drive sound calculation processing unit 209 outputs NC_Gain [0] to [511] representing the subtraction amount for each frequency spectrum and a drive noise detection signal.

  Although details will become clear from the description to be described later, the sensitivity difference correction unit 208 performs the Main [0] to [511] of the current frame in accordance with the driving noise detection signal one frame before from the driving sound calculation processing unit 209. The sensitivity of Sub [0] to [511] is corrected, and the corrected frequency spectrum data Main [0] to [511] and Sub [0] to [511] are output.

  The wind noise calculation processing unit 210 detects wind noise from the frequency spectrum data from the FFT unit 207 and determines a subtraction amount. Then, the wind noise calculation processing unit 210 outputs the determined wind noise frequency spectrum data WC_Gain [0] to [511] and the wind noise level signal.

  Stereo gain calculation processing section 211 determines the gains of stereo Lch (left channel) and Rch (right channel) for the frequency spectrum data from FFT section 207. Then, the stereo gain calculation processing unit 211 outputs Gain_L [0] to [511] and Gain_R [0] to [511] representing the gains of the determined frequency spectrum components of each channel. Here, the gain of the left channel is Gain_L [0] to [511], and the gain of the right channel is Gain_R [0] to [511].

The total gain calculation unit 212 includes NC_Gain [0] to [511], WC_Gain [0] to [511] determined by the drive sound calculation processing unit 209, the wind noise calculation processing unit 210, and the stereo gain calculation processing unit 211. Gain_L [0] to [511] and Gain_R [0] to [511] are added together to output Total_Gain_L [0] to [511] and Total_Gain_R [0] to [511]. Specifically, it is as follows. In the embodiment, the total gain calculation unit 212 functions as a total gain determination unit.
Total_Gain_R [] = NC_Gain [] + WC_Gain [] + Gain_R []
Total_Gain_L [] = NC_Gain [] + WC_Gain [] + Gain_L []

  The L / Rch generation unit 213 obtains the frequency spectrum for each frequency of MAIN [0] to [511], and Total_Gain_L [0] to [511] and Total_Gain_R [0] to [511] determined by the total gain calculation unit 212. By using this, frequency spectrum data of Lch and Rch is generated (details will be described later). That is, the L / Rch generation unit 213 in this embodiment functions as a stereo generation unit.

  The iFFT unit 214 performs inverse fast Fourier transform on the frequency spectrum data of each channel generated by the L / Rch generation unit 213, and returns the time-series audio signal of each channel.

  The audio processing unit 215 performs processing such as equalizer. The auto level controller adjusts the amplitude of the time-series audio signal to a predetermined level (hereinafter, ALC unit 216).

  With the above configuration, the audio input unit 102 performs predetermined processing on the audio signal to form audio data, and transmits the audio data to the memory 103.

  Next, a recording operation of the imaging apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 3 is a flowchart illustrating a recording sequence of the imaging apparatus 100 according to the embodiment.

  In S401, the recording (REC) start is instructed by the operation of the operation unit 110 by the user, and this processing is started. In step S402, the control unit 109 connects a voice path for voice recording. After the voice path is established, in step S403, the control unit 109 performs initial setting of signal processing including control described in the present embodiment, and starts processing. The contents of this signal processing will be described later. Thereafter, signal processing including control described in the present embodiment is performed until the REC sequence ends.

  During the recording processing sequence, the control unit 109 monitors an operation on the operation unit 110 by the user. When the zoom lever that is a part of the operation unit 110 is operated by the user, the process proceeds from S404 to S405, and the control unit 109 controls the imaging unit 101 to perform zoom processing. This zoom process continues until it is determined in S406 that the user has stopped operating the zoom lever. It should be noted that during the zoom process, as described above, driving noise is generated due to the movement of the lens 201, and the noise is recorded superimposed on the ambient environmental sound.

  If the control unit 109 determines that the end of recording is instructed by the operation of the operation unit 110 by the user or the status of the recording medium 108, the control unit 109 advances the process from S407 to S408. In step S408, the control unit 109 disconnects the voice path, and then ends the signal processing in step S409.

  Next, details of the audio input unit 102 of the imaging apparatus 100 of the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing a detailed configuration of the voice input unit 102 of the present embodiment.

  As described above, the audio input unit 102 according to the present embodiment includes the main microphone 205a and the sub microphone 205b that convert audio vibration propagating in the air into an electric signal and output the audio signal. As described above, the A / D conversion unit 206 performs sampling of the analog audio signal at 48 KHz and 16 bits, and converts the analog audio signal into digital audio data.

  The sensitivity difference correction unit 208 corrects the difference in sensitivity between the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b. For this reason, the sensitivity difference correction unit 208 includes a sensitivity correction integrator 2081, a sensitivity correction detection unit 2082, a correction amount calculation unit 2083, a sensitivity correction gain table 2084, and a sensitivity difference correction gain unit 2085.

  The sensitivity correction integrator 2081 changes the level in the time axis direction with respect to the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b. Has a time constant.

  The sensitivity correction detection unit 2082 has a level difference “Main [n]” between Main [0] to [511] and Sub [0] to [511] which is frequency spectrum data provided with a time constant by the sensitivity correction integrator 2081. -Sub [n] "is obtained for all frequency points. Here, it should be noted that the sign of the difference is generated.

  When the difference level from the sensitivity correction detection unit 2082 is negative (equivalent to Main [n] <Sub [n]), the correction amount calculation unit 2083 makes Main [n] = Sub [n]. Therefore, the correction amount of Sub [n] is calculated.

  When the difference level from the sensitivity correction detection unit 2082 is positive (equivalent to Main [n] ≧ Sub [n]), it is not necessary to correct Sub [n]. Therefore, in this case, the correction amount calculation unit 2083 outputs 0 as the correction amount of Sub [n].

  The sensitivity correction gain table 2084 stores specific correction amounts of the frequency spectra Sub [0] to [511] calculated by the correction amount calculation unit 2083.

  The sensitivity difference correction gain unit 2085 actually executes level correction of each frequency spectrum Sub [0] to [511] based on the sensitivity correction gain table 2084.

  Here, the above time constant is set to a unit of several tens of seconds because the purpose is to delay the follow-up of the sensitivity correction as much as possible. Further, when the sensitivity correction integrator 2081 receives a drive noise detection signal indicating detection of drive noise from the drive detection unit 2095 described later, the sensitivity correction integrator 2081 stops its operation. This is intended to eliminate integration during an unstable period when the optical lens 201 is driven.

  The above is the description of each processing unit constituting the sensitivity difference correction unit 208 in the embodiment. Next, the drive sound calculation processing unit 209 will be described.

  The driving sound calculation processing unit 209 subtracts the driving noise subtracting amount NC_Gain [0] from Main [0] to [511] and Sub [0] to [511] which are frequency spectrum data from the main microphone 205a and the sub microphone 205b. [511] is determined, and a drive noise detection signal indicating that the drive noise has been detected is output. For this reason, the drive sound calculation processing unit 209 includes an Mch-Sch calculation unit 2091, a drive noise removal gain calculation unit 2092, an hourly amplitude fluctuation detection unit 2093, an hourly phase fluctuation detection unit 2094, a drive detection unit 2095, and an interframe amplitude. A difference detection unit 2096 and a drive sound subtraction amount integrator 2097 are included.

  The Mch-Sch operation unit 2091 calculates a value obtained by subtracting the frequency spectrum data Sub [0] to [511] from the sub microphone 205b from the frequency spectrum data Main [0] to [511] from the main microphone 205a. Output as subtraction amount.

  However, if Main [n]> Sub [n] at the nth point of the frequency spectrum, the subtraction amount [n] is 0. That is, the Mch-Sch operation unit 2091 outputs a negative value as the subtraction amount [n] on the condition that Main [n] −Sub [n] <0 at the nth point of the frequency spectrum.

  In addition, when Sub [n] is sufficiently larger than Main [n] and Main [n] −Sub [n] falls below a preset threshold value (negative value), the Mch-Sch operation unit 2091 drives A detection signal [n] indicating that noise has been detected is output. If no, a detection signal is not output. Actually, noise detection may be expressed as “1” and non-detection as “0”.

  The determination of driving noise detection may be performed by comparing Sub [n] −Main [n] and a threshold value (having a positive value) with the subtraction relationship reversed. In this case, the Mch-Sch calculation unit 2091 outputs a signal indicating drive noise detection when the result of this calculation exceeds a threshold value.

  The drive detection unit 2095 receives the detection signals [0] to [511] for one frame from the Mch-Sch calculation unit 2091. A drive noise detection signal indicating that noise has been detected is output.

The processing by the Mch-Sch operation unit 2091 and the drive detection unit 2095 determines whether or not “i” (i is any of 0 to 511) that satisfies the following expression exists when the positive threshold is defined as Th. It can be said that the determination result is output as a signal indicating drive noise detection.
Main [i] + Th <Sub [i]

  The hourly amplitude fluctuation detection unit 2093 receives the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b between frames in the time direction. The amplitude fluctuation amount is detected. Specifically, the hourly amplitude fluctuation detection unit 2093 obtains and outputs a difference value between the n-th component value of the frequency spectrum of the current frame and the n-th component value of the frequency spectrum of the previous frame. . When the fluctuation amount at the n-th point exceeds a preset threshold value, the hourly amplitude fluctuation detection unit 2093 outputs the hourly amplitude fluctuation amount [n], and outputs 0 when it is equal to or smaller than the threshold value. To do.

  The hourly phase fluctuation detection unit 2094 is based on phase information acquired from a phase difference determination unit 2111 described later, and frequency spectrum data Main [0] to [511] from the main microphone 205a and frequency spectrum data Sub [ 0] to [511] are detected. For example, when the fluctuation amount exceeds a predetermined threshold at the nth point of the frequency spectrum, the hourly phase fluctuation detection unit 2094 outputs the hourly phase fluctuation amount [n]. When the fluctuation amount is equal to or smaller than the threshold value, the hourly phase fluctuation detection unit 2094 does not output the hourly phase fluctuation amount [n] or outputs the hourly phase fluctuation amount [n] = 0.

  Based on the drive noise detection signal from the drive detection unit 2095, the inter-frame amplitude difference detection unit 2096 detects the amplitude difference between frames in the time direction of Sub [0] to [511], which is the frequency spectrum data from the sub microphone 205b. Perform detection. For example, when there is a driving noise detection signal at the nth point of the frequency spectrum and the amplitude difference between the previous frame and the current frame exceeds a predetermined threshold, the interframe amplitude difference detection unit 2096 selects the interframe amplitude difference amount. [N] is output. When the difference is equal to or smaller than the threshold value, the inter-frame amplitude difference detection unit 2096 does not output the inter-frame amplitude difference amount [n] or outputs the inter-frame amplitude difference amount [n] = 0.

  In the same frame, the drive noise elimination gain calculation unit 2092 subtracts [0] to [511] from the above-described Mch-Sch calculation unit 2095, and the hourly amplitude fluctuation amount [0] from the hourly amplitude fluctuation detection unit 2093. To [511], hourly phase fluctuation amounts [0] to [511] from the hourly phase fluctuation detection unit 2094, and interframe amplitude difference amounts [0] to [511] from the interframe amplitude difference detection unit 2096, respectively. Is multiplied by a predetermined system number, and the added drive noise removal amounts [0] to [511] are calculated and output.

  The drive sound subtraction amount integrator 2097 gives a time constant to the amount of fluctuation in the time direction with respect to the drive noise removal amounts [0] to [511] output from the drive noise removal gain calculation unit 2092, thereby driving noise removal gain. NC_Gain [0] to [511] (with positive and negative signs) are output.

  The above is the configuration and operation of the drive sound calculation processing unit 209 of the embodiment. Next, the wind noise calculation processing unit 210 will be described.

  The wind noise calculation processing unit 210 detects wind noise from the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b, and calculates the subtraction amount. WC_Gain [0] to [511] and a wind noise level signal are output. The wind noise calculation processing unit 210 includes a wind detection unit 2101, a wind noise gain calculation unit 2102, and a wind noise subtraction amount integrator 2103.

The wind detection unit 2101 has a predetermined number of low frequency regions out of the frequency spectra Main [0] to [511] from the main microphone 205a and the frequency spectra Sub [0] to [511] from the sub microphone 205b. The wind noise level is detected according to the correlation of the points. For example, at 10 points in the low band, the wind noise level is obtained and output according to the following equation. Here, “n” is 0 to 9 in the embodiment, but this number may be changed as appropriate.
Wind noise level = Σ (Main [n] −Sub [n]) / (Main [n] + Sub [n])
Note that Σ in the above equation indicates the sum of n = 0 to 9.

  The wind noise gain calculation unit 2102 has a table having characteristic line segments as shown in FIG. As shown in the figure, one line segment has a negative gain below a certain frequency, and has a gain of zero above that frequency. Then, it includes a plurality of line segments having different frequency positions at which the gain changes from negative to zero. Then, the wind noise gain calculation unit 2102 determines and outputs wind noise gains [0] to [511] using one line segment according to the wind noise level. In the embodiment, the wind noise gains [0] to [511] are determined using a table. However, the wind noise gains [0] to [511] are determined using a function having the wind noise level as an argument. May be determined.

  The wind noise subtraction amount integrator 2103 gives a time constant to the fluctuation amount in the time direction with respect to the wind noise gains [0] to [511] output from the wind noise gain calculation unit 2102, and wind noise gain WC_Gain [0 ] To [511] (with positive and negative signs) are output.

  The above is the configuration and operation of the wind noise calculation processing unit 210 in the embodiment. Next, the stereo gain calculation processing unit 211 in the embodiment will be described.

  The stereo gain calculation processing unit 211 obtains the stereo Lch gain Gain_L [0] from the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b. ] To [511] and Rch gains Gain_R [0] to [511] are generated and output. For this purpose, the stereo gain calculation processing unit 211 includes a phase difference determination unit 2111, a stereo gain calculation unit 2112, a stereo suppression unit 2113, a left gain integrator 2114, and a right gain integrator 2115.

  The phase difference determination unit 2111 calculates the phase information of Sub [0] to [511] with respect to the frequency spectrum data Main [0] to [511].

For example, when the phase vector of each point in the frequency spectrum data is expressed as V (), the phase information [n] at the frequency point n is calculated according to the following equation.
Phase information [n] = | V (Main [n]) × V (Sub [n]) | / (| V (Main [n]) | · | V (Sub [n]) |)
Here, “| x |” on the right side represents the absolute value (scalar) of the vector x, “·” in the denominator represents the product of the scalars, and “x” in the numerator represents the outer product that is the sine of the two vectors. Yes.

  The phase difference determination unit 2111 outputs the phase information [0] to [511] calculated according to the above formula.

The stereo gain calculation unit 2112 calculates stereo gains [0] to [511] from the phase information [0] to [511] from the phase difference determination unit 2111. For example, at the frequency point n, the gain of each channel is obtained according to the following equation.
Stereo gain for Lch generation = 1 + phase information [n] × stereo gain for enhancement coefficient Rch = 1−phase information [n] × enhancement coefficient The stereo gain computing unit 2112 calculates Lch and Rch calculated by the above equation. Stereo gain [n] is output. Here, the enhancement coefficient is changed according to the frequency, and the upper limit is 1 and the lower limit is 0.

  The stereo suppression unit 2113 sets the enhancement coefficient to 0 when receiving a detection signal indicating that drive noise is detected from the Mch-Sch calculation unit 2091 in the drive sound calculation processing unit 209. Further, the stereo suppression unit 2113 sets the enhancement coefficient to 0 according to the wind noise level from the wind detection unit 2101 in the wind noise calculation processing unit 210.

  The left gain integrator 2114 gives a predetermined time constant to the amount of fluctuation in the time direction with respect to the stereo gain [0] to [511] for generating Lch output from the stereo gain calculation unit 2112, and converts the stereo gain into stereo. The gains are output as GainL [0] to [511] (with positive and negative signs).

  The right gain integrator 2115 gives the Rch generation stereo gain [0] to [511] output from the stereo gain calculation unit 2112 a predetermined time constant to the amount of fluctuation in the time direction, and converts it to the stereo gain. Output as gains GainR [0] to [511] (with positive and negative signs).

  The above is the configuration and operation of the stereo gain calculation processing unit 211 of the embodiment. Next, the total gain calculation unit 212 in the embodiment will be described.

The total gain calculation unit 212 includes NC_Gain [0] to [511], WC_Gain [0] to [511] determined by the drive sound calculation processing unit 209, the wind noise calculation processing unit 210, and the stereo gain calculation processing unit 211. Gain_L [0] to [511] and Gain_R [0] to [511] are added together to output Total_Gain_L [0] to [511] and Total_Gain_R [0] to [511]. Specifically,
Total_Gain_L [] = NC_Gain [] + WC_Gain [] + Gain_L []
Total_Gain_R [] = NC_Gain [] + WC_Gain [] + Gain_R []

  Next, the L / Rch generation unit 213 will be described. The L / Rch generator 213 uses Total_Gain_L [0] to [511] and Total_Gain_R [0] to [511] determined by the total gain calculator 212 from the frequency spectrum data MAIN [0] to [511]. , Lch and Rch output frequency spectrum data is created. The L / Rch generation unit 213 includes an Mch / Sch selection unit 2131 and an L / Rch gain addition unit 2132.

  The Mch / Sch selection unit 2131 is a range of frequency points of Sub [0] to [511] to be combined with the frequency spectrum data Main [0] to [511] according to the wind noise level by the wind detection unit 2101. Select. Further, the Mch / Sch selection unit 2131 changes the boundary position to be synthesized from the low frequency point to the high frequency point according to the wind noise level. When no wind is detected, the Mch / Sch selection unit 2131 does not perform synthesis and outputs the frequency spectrum data Main [0] to [511] as they are.

  The L / Rch gain addition unit 2132 applies the Total_Gain_L [0] to [511] determined by the total gain calculation unit 212 to the frequency spectrum data Main [0] to [511] output from the Mch / Sch selection unit 2132. , Total_Gain_R [0] to [511] are used to create frequency spectrum data of the left and right channels (Lch and Rch).

  The above is the configuration and operation of the L / Rch generation unit 213 of the embodiment.

  The iFFT unit 214 performs inverse transform (inverse FFT transform) on the frequency spectrum data of each channel generated by the L / Rch generation unit 213, and returns the original time-series audio signal. The audio processing unit 215 performs processing such as equalizer. An ALC (auto level controller) 216 adjusts the amplitude of the time-series audio signal to a predetermined level.

  With the above configuration, the voice input unit 102 performs predetermined processing on the voice signal to form voice data, and transmits the voice data to the memory 103 for storage.

  Here, a mechanical configuration constituting a part of the voice input unit 102 of the present embodiment will be described with reference to FIGS.

  FIG. 3A is an external view of the housing of the imaging apparatus according to the present embodiment. With the imaging device facing the object to be imaged, the reference symbol “a” at a predetermined position on the right side when viewed from the photographer is the input hole (opening) of the main microphone 205a, and the reference symbol “b” at the opposite position on the left side. This is an input hole for the sub microphone 205b. The enlarged view in FIG. 3B is a mechanical component of the main microphone 205a and the sub microphone 205b that are part of the audio input unit 102. FIG. 3B is a cross-sectional view showing the mechanical configuration. The exterior part 102-1 constituting the microphone hole, the main microphone bush 102-2a for holding the main microphone 205a, the sub microphone bush 102-2b for holding the sub microphone 205b, and pressing for pressing and holding the respective microphone bushes against the exterior part The unit 103 is configured. The exterior portion 102-1 and the pressing portion 103 are made of a mold member such as a PC material, but there is no problem even if it is a metal member such as aluminum or stainless steel. The main microphone bushing 102-2a and the sub microphone bushing 102-2b are made of a rubber material such as ethylene propylene diene rubber.

  Here, the diameter of the microphone hole in the exterior part will be described. The diameter (open area) of the microphone hole to the sub microphone 205b is smaller than the diameter (same area) of the microphone hole to the main microphone 205a and is reduced at a predetermined magnification. The microphone hole shape is preferably circular or elliptical, but may be rectangular. Moreover, about each hole shape, the same shape or another shape may be sufficient. The above configuration is intended to make it difficult for drive noise transmitted by air propagation to the microphone inside the imaging apparatus to be leaked from the microphone hole side of the sub microphone 205b to the outside.

  Next, the space in front of the microphone constituted by the exterior part 102-1 and the microphone bush will be described. The volume of the space in front of the sub microphone 205b composed of the exterior portion 102-1 and the sub microphone bush 102-2b is the volume of the space in front of the main microphone 205a composed of the exterior portion 102-1 and the main microphone bush 102-2a. It is larger than that and has a configuration that secures a volume of a predetermined magnification. The purpose of this configuration is to increase the atmospheric pressure change in the space in front of the sub microphone 205b and enhance the driving noise.

  As described above, the sub microphone 205b input in the microphone input mechanical configuration has a configuration in which the amplitude of the driving noise is greatly emphasized with respect to the main microphone 205a input. The relationship between the sound levels of the drive noises input to the microphones is main microphone 205a <sub microphone 205b. On the other hand, the level relationship of sound from outside the device (peripheral environmental sound that is the original purpose of sound collection) input to each microphone by air propagation from the front of the microphone hole is such that main microphone 205a ≧ sub microphone 205b. Please be careful.

  Here, the operation of the stereo gain calculation processing unit 211 in the audio input unit 102 of the present embodiment will be described with reference to FIGS.

  FIG. 7 shows an example of a sound path from the outside to the microphone built in the imaging apparatus 100 and a sound path when the built-in optical lens 201 is driven. The microphones at this time correspond to the main microphone 205a and the sub microphone 205b shown in FIG. As shown in FIG. 7, the distance between the sound source of the ambient environmental sound and the imaging device 100 is sufficiently larger than the distance between the main microphone 205a and the sub microphone 205b. Therefore, the sound propagation path from the ambient sound source to the main microphone 205a and the sound propagation path from the ambient sound source to the sub microphone 205b may be considered to be almost the same. However, the optical lens 201 built in the imaging apparatus is close to the main microphone 205a and the sub microphone 205b. In addition, there is a possibility that the distance from the motor to the microphone for moving the optical lens 201 is not uniform, or the sound path in the imaging apparatus is different. Therefore, the sound paths (distances) from the optical lens driving system to the main microphone 205a and the sub microphone 205b are greatly different. That is, there is a large difference in the difference between the sound levels of Mch and Sch between the ambient environmental sound and the driving noise. Therefore, there is a large difference between the ambient environmental sound and the driving noise of the optical lens, and these can be easily distinguished.

  On the other hand, it is difficult for Mch and Sch to determine from the left or right side whether the ambient environmental sound is originally generated. Therefore, the ambient environmental sound can be determined using the phase of the audio signal. Details will be described.

  FIGS. 8A to 8C show the relationship between certain frequency spectrum data Main [n] and Sub [n].

  The stereo gain calculation processing unit 211 obtains a stereo Lch gain Gain_L [0] from the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b. ] To [511] and Rch gains Gain_R [0] to [511] are output. The stereo gain calculation processing unit 211 has the following configuration.

  The phase difference determination unit 2111 calculates phase information of the frequency spectrum data Sub [0] to [511] with respect to the frequency spectrum data Main [0] to [511].

For example, when the ambient environmental sound at the frequency point n is generated from the main microphone 205a side, the relationship between V (Main [n]) and V (Sub [n]) is as shown in FIG. Even in the microphone arrangement in the present embodiment, the phase does not change even if the size of the frequency spectrum changes. Therefore, the phase information is obtained by using the outer product of V (Main [n]) and V (Sub [n]) (| V (Main [n]) × V (Sub [n]) |).
Phase information [n] = | V (Main [n]) × V (Sub [n]) | / (| V (Main [n]) |. | V (Sub [n]) |)
The phase difference determination unit 2111 outputs the phase information [n] calculated by the above equation. The phase information [n] obtained here is sin θ of V (Main [n]) and V (Sub [n]), and the ambient environmental sound is on the main microphone 205a side (the right side of the user holding the imaging device 100). ) 0 <phase information [n] ≦ 1.

  When the ambient environmental sound at the frequency point n is generated from the sub microphone 205b side, the relationship between V (Main [n]) and V (Sub [n]) is the frequency spectrum as shown in FIG. Become. Even in the microphone arrangement in the present embodiment, the phase does not change even if the size of the frequency spectrum changes.

Therefore, the phase information is obtained by using the outer product of V (Main [n]) and V (Sub [n]) (| V (Main [n]) × V (Sub [n]) |).
Phase information [n] = | V (Main [n]) × V (Sub [n]) | / (| V (Main [n]) |. | V (Sub [n]) |)
The phase difference determination unit 2111 outputs the phase information [n] calculated by the above equation. The phase information [n] obtained here is sin θ of V (Main [n]) and V (Sub [n]), and when the ambient environmental sound is from the sub microphone 205b side, 0> phase information [n ] ≧ −1.

  When the ambient environmental sound at the frequency point n is generated from the same distance as the main microphone 205a and the sub microphone 205b, that is, from the center of the optical lens 201, the relationship between V (Main [n]) and V (Sub [n]) is shown in FIG. The frequency spectrum relationship is as shown in (c). Even in the microphone arrangement in the present embodiment, the phase does not change even if the size of the frequency spectrum changes.

The phase information can be obtained by using the outer product (| V (Main [n]) × V (Sub [n]) |) of V (Main [n]) and V (Sub [n]).
Phase information [n] = | V (Main [n]) × V (Sub [n]) | / (| V (Main [n]) |. | V (Sub [n]) |)
The phase difference determination unit 2111 outputs the phase information [n] calculated by the above equation. The phase information [n] obtained here is sin θ of V (Main [n]) and V (Sub [n]), and the ambient sound is phase information [n] ≈0 from the sub microphone 205b side. .

The stereo gain calculation unit 2112 calculates the stereo gains [0] to [511] using the phase information [0] to [511] determined as described above. For example, at the frequency point n, the stereo gain calculator 2112 calculates the gain of each channel according to the following equation.
Stereo gain for Lch generation = 1 + phase information [n] × enhancement coefficient Rch generation stereo gain = 1−phase information [n] × enhancement coefficient Further, the stereo gain calculation unit 2112 calculates each of the above-described equations. The stereo gain [n] of the channel is output.

  FIG. 9 is a diagram showing enhancement coefficients at each frequency point used in the stereo gain calculator 2112.

  When the horizontal axis is the frequency point and the vertical axis is the emphasis coefficient, the emphasis coefficient of the frequency to be most emphasized is 1.0 as the maximum value. The minimum value is 0.

  For example, the emphasis coefficient is 1.0 for 1 kHz to 5 kHz that is most emphasized, and 0 for 200 Hz or less.

  The high-frequency enhancement coefficient for which the phase difference cannot be determined is determined by the distance between the main microphone 205a and the sub microphone 205b. For example, when the distance between the main microphone 205a and the sub microphone 205b is 15 mm, if the sound speed is 340 m / s, the correct phase information cannot be obtained and the left and right are reversed when the half-wavelength is 11.3 kHz or more between 15 mm. There is a possibility. In addition, the accuracy is low at 5.7 kHz or more where a quarter wavelength enters between 15 mm. Therefore, the emphasis coefficient is applied according to the frequency as shown in FIG.

  Here, operations of the drive sound calculation processing unit 209, the total gain calculation unit 212, and the L / Rch generation unit 213 in the voice input unit 102 of the present embodiment will be described with reference to FIGS. 5 and 10 to 13.

  FIG. 10 shows an example of amplitude spectrum data of each frequency of the main microphone 205a and the sub microphone 205b.

  The FFT unit 207 converts the audio signal of each channel as a frequency spectrum of 1024 points from 0 Hz to 48 kHz. The converted frequency spectrum data is assumed to have a 512-point frequency spectrum up to 24 kHz which is the Nyquist frequency.

As described above with reference to FIGS. 3A and 3B, according to the microphone input mechanical configuration of the imaging apparatus 100 according to the embodiment, the sub microphone 205b has an amplitude of drive noise relative to the main microphone 205a. Produces a greatly enhanced signal. In other words, in the amplitude spectrum,
Ambient environmental sound level: main microphone 205a ≧ sub microphone 205b
Driving noise level: main microphone 205a <sub microphone 205b
It becomes the relationship.

  FIG. 10 shows an example of amplitude spectrum data Main [] from the main microphone 205a and amplitude spectrum data Sub [] from the sub microphone 205b. Further, “Main-Sub” in the figure indicates subtraction amounts [0] to [511] calculated by the Mch-Sch operation unit 2091 by subtracting Sub [] from Main [].

  For example, focusing on the amplitude spectrum around the Nth point in Sch, it can be said that Sch> Mch, that is, the driving noise is the dominant point. At this time, in Main-Sub, a subtraction amount that exceeds (below) a predetermined zoom detection threshold around the Nth point is calculated, and an amplitude spectrum that is “driving noise” is detected around the Nth point. The On the other hand, paying attention to the amplitude spectrum at the N2th point in Mch, Sch ≦ Mch. In other words, it can be said that ambient sound is the dominant point. At this time, since the subtraction amount exceeding the zoom detection threshold is not calculated in Main-Sub, the amplitude spectrum around the N2th point is not detected as drive noise. The above calculation is performed in the entire range of the amplitude spectrum of [0] to [511].

  FIG. 11 is a diagram showing a time-series amplitude spectrum at the frequency N point of the sub microphone 205b.

  “Sub ch” in the figure indicates that the amplitude spectrum data of the Nth point fluctuates in time series.

_{Sch | t n -t (n-} 1) | , compared amplitude spectrum of SchN point th represents the amplitude change amount between the time direction of the frame is calculated by the time each amplitude variation detecting section 2093, each time variation Output as quantity [n]. For example, when attention is focused on the amplitude spectrum of Sch from t1 to t2, the amount of fluctuation in the time direction is large, and the amount of fluctuation is detected from t1 to t2 at Sch | t _n −t _(n−1) |. The amount of hourly fluctuation exceeding the threshold is calculated. This calculation is executed at all points in the amplitude spectrum of [0] to [511].

  FIGS. 12A and 12B are diagrams showing time-series phases at the Nth frequency in the amplitude spectrum from the main microphone 205a and the amplitude spectrum from the sub microphone 205b.

  FIG. 6A shows changes in the phase of the “ambient environmental sound” in the time direction by the complex planes Im and Re. The solid line portion represents Mch and the dotted line portion represents Sch. About t0, t1, t2, t3, t4, the transition of a time direction is shown.

  FIG. 4B shows the phase change of “driving noise”.

  Here, for ambient sound, the phases of Mch and Sch are constant over time from t0 to t4. Regarding the driving noise, the phases of Mch and Sch greatly fluctuate in the transition of time from t0 to t4. The phase fluctuation in each time direction is detected by the hourly phase fluctuation detection unit 2094 and output as the hourly phase fluctuation amount [n]. The hourly phase fluctuation detection unit 2094 executes this calculation for all frequency points in the amplitude spectrum of [0] to [511].

  FIGS. 13A and 13B show an example of an operation timing chart of the Mch-Sch operation unit 2091. FIG.

  Main [N], Sub [N], and Main [N] -Sub [N] in FIG. 4A are Mch amplitude spectrum data, Sch amplitude spectrum data, and Mch amplitude spectrum at the frequency N point, respectively. The subtraction amount [N] obtained by subtracting the amplitude spectrum is shown. Main [N] -Sub [N] outputs the result of the operation performed by the Mch-Sch operation unit 2091.

  Here, focusing on the period from t1 to t2 in FIG. 9A, the amplitude spectrum of Sub [N] is significantly higher than that of Main [N], and the calculation of Main [N] −Sub [N] is performed. The result exceeds the zoom threshold, and is detected as drive noise, and a subtraction amount [N] is output.

  Main [N2], Sub [N2], and Main [N2] -Sub [N2] in FIG. 13B are the Mch amplitude spectrum, the Sch amplitude spectrum, and the Mch amplitude spectrum at the frequency N2 point, respectively. Represents the subtraction amount [n]. Here, focusing on the period from t1 to t2 in FIG. 5B, Main [N2] and Sub [N2] fluctuate at the same level, and the calculation result of Main [N2] -Sub [N2] is also zoomed. There is no result exceeding the threshold. The driving noise is not detected at the frequency N2 point. The Mch-Sch operation unit 2091 performs the operation shown in the timing chart on all the amplitude spectra [0] to [511].

  FIG. 5 shows an example of a timing chart of the L / Rch generation unit 213. The zoom driving operation is controlled by the control unit 109, and the optical lens 201 is driven at a timing from t1 to t2. The Mch spectrum represents the spectrum of the specific frequency N point extracted in FIG. Lch and Rch are generated by adding Total_Gain_L and Total_Gain_R determined by the total gain calculation unit 212 to Mch. As shown in the timing chart of the figure, for example, by lowering Total_Gain_L and raising Total_Gain_R with respect to Mch, Rch can be emphasized, and 2ch stereo signals can be generated with 1ch input. is there.

  Further, even during the driving operation of the optical lens from t1 to t2, it is possible to remove drive noise from Lch and Rch by lowering Total_Gain_L and Total_Gain_R.

  Here, the operation of the sensitivity difference correction unit 208 in the voice input unit 102 of the present embodiment will be described with reference to FIG.

  FIG. 14 shows an example of an operation timing chart of the sensitivity difference correction unit 208. In the figure, the zoom detection indicates the detection result of the drive noise of the drive detection unit 2095. The input spectrum NPoint indicates the Mch amplitude spectrum and the Sch amplitude spectrum at the Nth frequency point. The solid line portion indicates Mch, and the dotted line portion indicates Sch.

  Input spectrum (integration) NPoint indicates the integration result of Mch and Sch of the sensitivity correction integrator 2081 at the frequency N point. The sensitivity adjustment output spectrum NPoint indicates the Mch amplitude spectrum and the Sch amplitude spectrum level-corrected by the sensitivity difference correction gain unit 2085 at the frequency N point. The solid line portion indicates Mch, and the dotted line portion indicates Sch.

  In FIG. 14, t0 is the REC start timing, and represents a sufficiently long time of about several tens of seconds from t0 to t1. From timing t2 to t3, the zoom detection is ON, and the drive detection unit 2095 indicates that drive noise is generated.

  In the input spectrum NPoint, there is a level difference between Mch and Sch at the start of REC t0. On the other hand, the input spectrum (integration) NPoint is integrated by the sensitivity correction integrator 2081 and slowly follows the level difference from t0 to t1. The sensitivity adjustment output spectrum NPoint is also subjected to gain correction by the sensitivity difference correction gain unit 2085 with sufficient time from t0 to t1 with respect to the integration result of the input spectrum (integration) NPoint. This is because the sensitivity difference correction unit 208 is intended to correct the sensitivity of the main microphone 205a and the sub microphone 205b. Therefore, level correction over a sufficient time of about several tens of seconds is sufficient, and transient response is required. do not do.

  Further, in the zoom detection ON period from timing t2 to t3, the sensitivity correction integrator 2081 is stopped. Therefore, a large level difference occurs between the amplitude spectrum of Mch and the amplitude spectrum of Sch due to the generation of driving noise. However, since the sensitivity correction integrator 2081 is in a stopped state, the value does not follow the level difference. Retained. As described above, the sensitivity difference correction unit 208 is intended to correct the sensitivity of the main microphone 205a and the sub microphone 205b, and therefore does not require a response to a transient level difference due to driving noise. The sensitivity difference correction unit 208 performs the correction shown in the timing chart on all the amplitude spectra [0] to [511].

  Here, the operation of the wind noise calculation processing unit 210 in the voice input unit 102 of the present embodiment will be described with reference to FIGS. 15 to 17.

  FIG. 15 is a cross-sectional view illustrating a mechanical configuration in which the windshield material 102-3 is configured for the sub microphone 205b which is a part of the voice input unit 102.

  The exterior part 102-1 constituting the microphone hole presses and holds the main microphone bush 102-2a for holding the main microphone 205a, the sub microphone bush 102-2b for holding the sub microphone 205b, and the respective microphone bushes against the exterior part. The pressing unit 103 is configured. The exterior portion 102-1 and the pressing portion 103 are made of a mold member such as a PC material, but there is no problem even if it is a metal member such as aluminum or stainless steel. The main microphone bushing 102-2a and the sub microphone bushing 102-2b are made of a rubber material such as ethylene propylene diene rubber.

  Here, the hole diameter of the microphone hole in the exterior part 102-1 will be described. The diameter of the microphone hole to the sub microphone 205b is smaller than the diameter of the microphone hole to the main microphone 205a. In the embodiment, the diameter (diameter) of the microphone hole of the sub microphone 205b is set to 1/3 of that of the microphone hole of the main microphone 205a. The microphone hole shape is preferably circular or elliptical, but may be rectangular. Moreover, about each hole shape, the same shape or another shape may be sufficient.

  Next, the space in front of the microphone constituted by the exterior portion 102-1 and the microphone bushes 102-2a and 102-2b and the arrangement of the cushion material will be described. The volume of the space in front of the sub microphone 205b composed of the exterior portion 102-1 and the sub microphone bush 102-2b is that of the space in front of the main microphone 205a composed of the exterior portion 102-1 and the main microphone bush 102-2a. Larger and 3 times assured.

  In the space in front of the sub microphone 205b configured by the exterior portion 102-1 and the sub microphone bush 102-2b, a windshield cushion material and a seal microphone are disposed as the windshield material 102-3. Both are configured as members that filter signal components in the low frequency band of about 0 to 4 kHz corresponding to the wind frequency. By the windshield material 102-3, it is possible to greatly reduce the influence of air propagation to the sub microphone 205b of wind noise in which the low frequency band is dominant.

FIG. 16 shows the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b when wind noise is input. When wind noise is input, the wind noise component exists in the low frequency band of the dotted line. The wind detection unit 2101 correlates, for example, 10 points in the low frequency band from the frequency spectrum Main [0] to [511] from the main microphone 205a and the frequency spectrum Sub [0] to [511] from the sub microphone 205b. The wind noise level is detected. The wind detection unit 2101 calculates and outputs a wind noise level according to the following equation, for example, at a low frequency point n.
Wind noise level = Σ (Main [n] −Sub [n]) / (Main [n] + Sub [n])
In the above equation, 10 points of low frequency components are used, and n is in the range of 0 to 9. In the embodiment, the low frequency band is 10 points, but this number is an example. It is desirable to set appropriately according to the design of the imaging device.

  FIG. 17 shows the frequency relationship of the wind noise gain [0] to [511] with respect to the wind noise level from the wind detection unit 2101 calculated by the wind noise gain calculation unit 2102. As the wind noise level from the wind detector 2101 increases, the wind noise gain shifts to the minus side, and the cutoff frequency indicated by the dotted line shifts to the high frequency band. Wind noise gains [0] to [511] are determined by the cut-off frequency.

  Next, the operation of the Mch / Sch selection unit 213 in the voice input unit 102 of the present embodiment will be described with reference to FIGS. 18 (a) and 18 (b).

  FIG. 18A shows frequency spectrum data Main [0] to [511] (Main channel in the figure) from the main microphone 205a and frequency spectrum data Sub from the sub microphone 205b synthesized by the Mch / Sch selection unit 2131. The relationship between the synthesis ratio according to the wind noise level and the frequency with [0] to [511] (Sub ch in the drawing) is shown.

  Here, FIG. 3A shows an embodiment corresponding to the mechanical configuration of the main microphone 205a and the sub microphone 205b shown in FIG. Here, as shown in FIG. 18A, the Mch / Sch selection unit 2131 sets Mainch at a ratio of 1.0 to 0.5 and Subch from 0 to 0.5 based on the wind noise level. Synthesize with the ratio of

  The higher the wind noise level, the lower the synthesis ratio from 1.0 to 0.5 at Main ch, and the synthesis ratio from 0 to 0.5 at Sub ch, and the crossover of synthesizing Main ch and Sub ch. Increase the frequency (upper limit frequency of synthesis). Then, the Mch / Sch selection unit 2131 synthesizes Main ch and Sub ch at a ratio below the upper limit frequency depending on the wind noise level, and selects and outputs the Main ch at a frequency exceeding the upper limit frequency. When the wind noise level is 0, the combination ratio of Sch is 0. Here, as described in FIG. 3B, the diameter of the microphone hole to the sub microphone 205b is smaller than the diameter of the microphone hole to the main microphone 205a and is reduced to 1/3. Therefore, the influence of wind noise on the sub microphone 205b is weaker than that of the main microphone 205a. Therefore, by combining Sch with Mch according to the wind noise level from the wind detection unit 2101, the effect of reducing wind noise is achieved.

  Next, FIG.18 (b) shows embodiment corresponding to the mechanical structure which comprised the windshield material 102-3 with respect to the sub microphone 205b as shown in FIG. Here, the Mch / Sch selection unit 2131 synthesizes Mch at a ratio of 1.0 to 0 and Sch at a ratio of 0 to 1.0 from the wind noise level. In other words, the greater the wind noise level, the lower the synthesis ratio from 1.0 to 0 for Mch, the synthesis ratio from 0 to 1.0 for Sch, and the crossover frequency for synthesizing Mch and Sch. Go. When the wind noise level is 0, the combination ratio of Sch is 0. Here, as described in FIGS. 3A and 15, the diameter of the microphone hole to the sub microphone 205b is smaller than the diameter of the microphone hole to the main microphone 205a and is reduced to 1/3. In addition, a windshield material 102-3 is provided in a space in front of the sub microphone 205b configured by the exterior portion 102-1 and the sub microphone bush 102-2b. Therefore, the influence of wind noise on the sub microphone 205b can be further reduced with respect to the main microphone 205a. Therefore, switching from Mch to Sch according to the wind noise level from the wind detection unit 2101 is effective in reducing wind noise.

  Here, a specific operation of the stereo suppression unit 2113 in the voice input unit 102 of the present embodiment will be described with reference to FIGS. 19 and 20.

  FIG. 19 shows a timing chart for changing the enhancement coefficient used for enhancing the stereo effect in the stereo suppression unit 2113 according to the drive noise detection and the wind noise detection. In FIG. 19, Main [N] indicates Mch amplitude spectrum data at the frequency N point. The drive noise detection signal is a detection signal indicating that the drive noise is detected by the drive detection unit 2095. The wind noise detection signal indicates a wind noise level (wind noise level equal to or higher than a preset threshold) indicating that the wind detection unit 2101 has detected wind noise. GainL [N] and GainR [N] indicate the stereo Lch and Rch gains to be added to the Mch amplitude spectrum at the Nth frequency determined by the stereo gain calculation processing unit 2112.

  The stereo suppression unit 2113 receives the detection signal indicating that the driving noise is detected from the Mch-Sch calculation unit 2091 and sets the enhancement coefficient to 0. Further, in response to a wind noise level indicating that wind noise has been detected from the wind detection unit 2101, the enhancement coefficient is set to 0 according to the frequency.

  Here, focusing on the period from the timing t1 to the timing t2, the amplitude spectrum of Main [N] greatly fluctuates, and the detection signal from the Mch-Sch calculation unit 2091 indicates detection as having drive noise. During this period, GainL [N] and GainR [N] are fixed to 0. In other words, this indicates that the stereo suppression unit 2113 has set the enhancement coefficient to 0. When attention is paid to the period from timing t3 to t4, the amplitude spectrum of Main [N] largely fluctuates, and the wind noise detection signal from the wind detection unit 2101 indicates detection. During this period, GainL [N] and GainR [N] are fixed to 0. In other words, this indicates that the stereo suppression unit 2113 has set the enhancement coefficient to 0.

  FIG. 20 shows the frequency spectrum Main [0] to [511] from the main microphone 205a and the frequency spectrum Sub [0] from the sub microphone 205b in the Mch / Sch selection unit 2131 when the wind noise level from the wind detection unit 2101 is detected. ] To [511] are diagrams showing the relationship between the ratio of combining frequency and the frequency at which the stereo suppression unit 2113 sets the enhancement coefficient to 0 with respect to the frequency. Here, the Mch / Sch selection unit 2131 decreases the synthesis ratio from 1.0 to 0.5 for Mch and increases the synthesis ratio from 0 to 0.5 for Sch as the wind noise level increases from the wind noise level. The frequency of the crossover for synthesizing Mch and Sch is increased. For wind noise levels, the crossover frequency is 500 Hz. On the other hand, the stereo suppression unit 2113 fixes the enhancement coefficient to 0 until a frequency 750 Hz higher than the crossover frequency. The stereo suppression unit 2113 increases the frequency at which the enhancement coefficient is fixed to 0 as the wind noise level from the wind detection unit 2101 increases. By emphasizing GainL and GainR with stereo gain, wind noise is also prevented from being enhanced.

  Here, operations of the drive sound subtraction amount integrator 2097, the wind noise subtraction amount integrator 2103, the right gain integrator 2114, and the left gain integrator 2115 in the sound input unit 102 of the present embodiment will be described with reference to FIG. .

  FIG. 21 shows the determined drive noise elimination gain NC_GAIN [N], wind noise subtraction amount WC_GAIN [N], Lch generation stereo gain L_GAIN [N], Rch for the Mch amplitude spectrum data at the Nth frequency point. A time constant for each of the generating stereo gains R_GAIN [N] is shown. These are determined by a driving sound subtraction amount integrator 2097, a wind noise subtraction amount integrator 2103, a left gain integrator 2114, and a right gain integrator 2115. The time constant of the driving noise subtraction amount integrator is slower than the time constant of the right gain integrator 2115 and the left gain integrator 2114, and the time constant of the wind noise subtraction amount integrator is the right gain integrator 2115 and the left gain integrator. Slow for 2114 time constant. Driving noise and wind noise are driving noise components, respectively, and have large variations in time series. The variation is suppressed by delaying the time constant and delaying the tracking of driving noise subtraction and wind noise subtraction, respectively. As for the stereo gain, the follow-up to the movement of the sounding subject is accelerated by increasing the time constant.

  In the present embodiment, the case where two lines of audio are input has been described, but the present invention can be applied even when the number of channels is more than that.

  In the present embodiment, the imaging apparatus has been described. However, the audio processing of the audio input unit 102 according to the present embodiment is any apparatus that records or inputs external audio, that is, an audio recording apparatus. Even a simple device can be applied. For example, you may apply to an IC recorder, a mobile telephone, etc.

  In the embodiment, the example in which the configuration illustrated in FIG. 6 is realized by hardware has been described. However, for example, procedures, subroutines, and the like that are executed by the processor in many of the processing units other than the microphone and the AD conversion unit in FIG. This program may be realized.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 ... Imaging device 101 ... Imaging part 102 ... Audio | voice input part 103 ... Memory 104 ... Display control part 105 ... Display part 106 ... Encoding process part 107 ... Recording / reproducing part 108 ... Recording medium, 109 DESCRIPTION OF SYMBOLS Control part 110 ... Operation part 111 ... Audio | voice output part 112 ... Speaker 113 ... External output part 114 ... Data bus 201 ... Optical lens 202 ... Imaging element 203 ... Image processing part 204 ... Optical lens Control unit, 205 ... microphone, 205a ... main microphone, 205b ... sub microphone, 206 ... A / D conversion unit, 207 ... FFT unit, 208 ... sensitivity difference correction unit, 209 ... drive sound calculation processing unit, 210 ... wind noise calculation processing 211, stereo gain calculation processing unit, 212 ... total gain calculation unit, 213 ... L / Rch generation unit, 214 ... iFFT unit, 215 ... audio processing unit, 16 ... ALC unit, 102-1 ... exterior, 102-2A ... main microphone bush, 102-2B ... sub microphone bush, 102-3 ... windshield member

Claims (7)

A housing,
A drive unit;
A first microphone housed in the housing such that sound is propagated through a first opening provided at a first predetermined position of the housing;
The sound is propagated through a second opening having a smaller area than the first opening provided at a second predetermined position of the housing related to the first predetermined position. 2, wherein the volume of the second space between the second microphone and the second opening is the first space between the first microphone and the first opening. The second microphone housed inside the housing to be larger than the volume of
Conversion means for converting time-series audio data obtained from the first microphone into first frequency spectrum data and time-series audio data obtained from the second microphone into second frequency spectrum data; ,
Calculating means for calculating the amount of noise by the driving unit for each frequency from the first frequency spectrum data and the second frequency spectrum data obtained by the converting means;
Based on the first frequency spectrum data, the second frequency spectrum data, and the amount of the noise calculated by the calculation means, the frequency spectrum data of the left channel in which the noise is suppressed, and the right channel Generating means for generating frequency spectrum data;
An audio processing apparatus comprising: inverse conversion means for inversely converting the frequency spectrum data of the left and right channels generated by the generation means into the audio data of the time-series left and right channels. A first microphone bush for holding the first microphone;
A second microphone bush for holding the second microphone;
The first space is constituted by the casing and the first microphone bush,
The audio processing apparatus according to claim 1, wherein the second space includes the casing and the second microphone bush.   The noise propagated to the second microphone via the second space is larger than the noise propagated to the first microphone via the first space. The speech processing apparatus according to 1.   The audio processing apparatus according to claim 1, wherein the first microphone is a microphone corresponding to one of a left channel and a right channel, and the second microphone is a microphone corresponding to the other.   The generation means determines the gain of each of the right channel and the left channel based on the first frequency spectrum data, the second frequency spectrum data, and the amount of noise calculated by the calculation means, The first frequency spectrum data is controlled by the right channel gain to generate the right channel frequency spectrum data, and the first frequency spectrum data is controlled by the left channel gain to control the left channel frequency spectrum data. The speech processing apparatus according to claim 1, wherein: A method for controlling a speech processing apparatus,
The sound processing device is accommodated in the housing so that sound is propagated through a housing, a drive unit, and a first opening provided at a first predetermined position of the housing. The first microphone and a second opening of the casing that is provided at a second predetermined position related to the first predetermined position and having a smaller area than the first opening. A second microphone through which sound is propagated, wherein a volume of a second space between the second microphone and the second opening is determined by the first microphone and the first opening. The second microphone housed in the housing to be larger than the volume of the first space between
The method
A conversion step of converting time-series audio data obtained from the first microphone into first frequency spectrum data, and converting time-series audio data obtained from the second microphone into second frequency spectrum data; ,
From the first frequency spectrum data and the second frequency spectrum data obtained by the conversion step, a calculation step for calculating the amount of noise by the driving unit for each frequency;
Based on the first frequency spectrum data, the second frequency spectrum data, and the amount of the noise calculated by the calculation step, the frequency spectrum data of the left channel in which the noise is suppressed, and the right channel Generating step of generating frequency spectrum data;
And a reverse conversion step of reversely converting the frequency spectrum data of the left and right channels generated in the generation step into the audio data of the time-series left and right channels, respectively. A program that is read and executed by a processor of a sound processing device,
The voice processing device
The sound processing device is accommodated in the housing so that sound is propagated through a housing, a drive unit, and a first opening provided at a first predetermined position of the housing. The first microphone and a second opening of the casing that is provided at a second predetermined position related to the first predetermined position and having a smaller area than the first opening. A second microphone through which sound is propagated, wherein a volume of a second space between the second microphone and the second opening is determined by the first microphone and the first opening. The second microphone housed in the housing to be larger than the volume of the first space between
The said program is a program for making the said processor perform each step of the method of Claim 6.

Images