JP6931296B2 - Speech processing device and its control method - Google Patents

Description

The present invention relates to a technique for emphasizing the stereo feeling of recorded voice.

An imaging device represented by a digital camera or a video camera can image a moving subject, record the data of the moving image obtained as a result, and also record the sound around the subject. Hereinafter, the sound around the subject to be recorded is hereinafter referred to as "ambient environment sound".

Japanese Unexamined Patent Publication No. 2010-173062

As a method of expressing the stereo feeling of the recorded voice, a method of enhancing the stereo feeling disclosed in Patent Document 1 is known. According to Patent Document 1, the position of the sound source is assumed from the sounds input from the left and right microphones, and the stereo feeling is emphasized by a method of controlling the level and time of the left and right sounds according to the positions. First, the phase difference is calculated for each frequency band with respect to the voice input from the left and right microphones, and the expected position of the sound source is obtained. For example, if it is determined that the sound source is on the right side, control is performed to increase the sound level input from the microphone on the right side. Alternatively, it is possible to emphasize the stereo feeling by delaying the sound input from the left microphone. However, in the method of emphasizing the stereo feeling as shown in Patent Document 1, when there is a difference in the level of the sound recorded by the left and right microphones such as the holding configuration of the left and right microphones is different, an appropriate stereo sound is generated. I can't.

The present invention has been made in view of such a problem, and provides a technique for obtaining a stereo audio signal that appropriately emphasizes the stereo feeling even when there is a difference in sound recorded by the left and right microphones that function as stereo microphones. It is something to try.

In order to solve this problem, for example, the voice processing device of the present invention has the following configuration. That is,
It is a voice processing device
With the first microphone
With the second microphone
Conversion that converts the time-series audio data obtained from the first microphone into the first frequency spectrum data and converts the time-series audio data obtained from the second microphone into the second frequency spectrum data. Means and
As a determination means for obtaining the phase shift of the second frequency spectrum data with respect to the first frequency spectrum data obtained by the conversion means and determining the gain of the right channel and the gain of the left channel based on the phase shift. ,
The first frequency spectrum data is controlled by the gain of the right channel determined by the determination means to generate the frequency spectrum data of the right channel, and the first frequency spectrum data is determined by the determination means. A generation means that is controlled by the gain of the left channel to generate frequency spectrum data of the left channel,
It has an inverse transformation means for inversely converting the frequency spectrum data of each of the left and right channels generated by the generation means into the audio data of each of the left and right channels in the time series.
The determining means is characterized in that when a predetermined noise is detected, the gain of the right channel and the gain of the left channel are determined without being based on the phase shift .

According to the present invention, it is possible to obtain a stereo sound that appropriately emphasizes the stereo feeling even when there is a difference in the sound recorded by the left and right microphones.

The block block diagram of the image pickup apparatus of an embodiment. A detailed block configuration diagram of an image pickup unit and an audio input unit of the image pickup apparatus of the embodiment. The mechanical block diagram of the audio input part of the image pickup apparatus of embodiment. The flowchart which shows the REC sequence of the image pickup apparatus of embodiment. The timing chart of the L / Rch generation part of the image pickup apparatus of embodiment. The block diagram which shows the detailed structure of the audio input part of the image pickup apparatus of embodiment. The figure which shows the system which propagates the ambient sound to an image pickup apparatus. The figure which shows the phase relationship of the frequency spectrum from the main microphone a and the frequency spectrum from a sub microphone b of the image pickup apparatus of an embodiment. The figure which shows the relationship between the emphasis coefficient of the stereo feeling of embodiment, and a frequency. The figure which shows the amplitude spectrum for each frequency of each of the main microphone a and the sub microphone b of the image pickup apparatus of an embodiment. The figure which shows the amplitude spectrum of the time series of the frequency N point of the sub microphone b of the image pickup apparatus of an embodiment. It is a figure which shows the phase of each of the main microphone a and the sub microphone b of the image pickup apparatus of an embodiment in time series. The operation timing chart of the Mch-Sch calculation unit of the image pickup apparatus of the embodiment. The operation timing chart of the sensitivity difference correction part of the image pickup apparatus of embodiment. The mechanical block diagram of the audio input part of the image pickup apparatus of embodiment. The figure which shows the frequency spectrum from the main microphone a and the frequency spectrum from a sub microphone b of the image pickup apparatus of an embodiment. The figure which shows the frequency relation of the wind noise gain with respect to the wind noise level of an embodiment. The figure which shows the relationship between the combined ratio and the frequency of the frequency spectrum from the main microphone a and the frequency spectrum from a sub microphone b of the image pickup apparatus of an embodiment. A timing chart for changing the emphasis coefficient used for enhancing the stereo effect according to the drive noise detection and the wind noise detection for the stereo suppression unit of the image pickup apparatus of the embodiment. The figure which shows the relationship between the synthesis ratio, the frequency, and the emphasis coefficient used for the enhancement of a stereo effect at the time of the wind noise detection of an embodiment. The figure which shows the time constant of the driving noise removal gain, the wind noise subtraction amount, the stereo gain for Lch generation, and the stereo gain for Rch generation of an embodiment.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings. In the present embodiment, the audio processing device accommodated in the image pickup device will be described.

FIG. 1 is a block diagram showing the configuration of the image pickup apparatus 100 of the embodiment. The image pickup apparatus 100 includes an image pickup unit 101, an audio input unit 102, a memory 103, a display control unit 104, and a display unit 105. Further, the image pickup apparatus 100 includes a coding processing unit 106, a recording / playback unit 107, a recording medium 108, a control unit 109, an operation unit 110, an audio output unit 111, a speaker 112, an external output unit 113, and a bus connecting them. It has 114.

The image pickup unit 101 converts the optical image of the subject captured by the photographing optical lens into an image signal by the image pickup element, performs analog-digital conversion, image adjustment processing, and the like to generate image data. The photographing optical lens may be a built-in optical lens or a detachable optical lens. The image sensor may be a photoelectric conversion element typified by a CCD, CMOS, or the like.

The voice input unit 102 collects peripheral sounds from outside the voice processing device (outside the imaging device in the embodiment) by a built-in microphone or a microphone connected via a voice terminal, and generates an electric signal. In addition, the voice input unit 102 performs analog-to-digital conversion, voice processing, and the like to generate voice data. The microphone may be directional or omnidirectional, but in the present embodiment, an omnidirectional microphone is used.

The memory 103 is used to temporarily store the image data obtained by the imaging unit 101 and the voice data obtained by the voice input unit 102.

The display control unit 104 displays an image related to the image data obtained by the image pickup unit 101, an operation screen of the image pickup device 100, a menu screen, and the like on an external display via the display unit 105 and a video terminal (not shown). .. The type of the display unit 105 is not limited, but is, for example, a liquid crystal display.

The coding processing unit 106 reads out the image data and audio data temporarily stored in the memory 103, performs predetermined coding, and generates compressed image data, compressed audio data, and the like. Further, the audio data may not be compressed. The compressed image data can be, for example, MPEG2 or H.M. It may be compressed by any compression method such as 264 / MPEG4-AVC. Further, the compressed audio data may also be compressed by a compression method such as AC3 (A) AC, ATRAC, ADPCM, or the like. The coding processing unit 106 also performs decoding processing of the above-mentioned coded data (compressed image data, compressed audio data).

The recording / playback unit 107 records compressed image data, compressed audio data or audio data, and various types of data generated by the coding processing unit 106 on the recording medium 108, and reads the data from the recording medium 108. Here, the recording medium 108 is a non-volatile recording medium for recording image data, audio data, and the like. For example, it may be a magnetic disk, an optical disk, a semiconductor memory, or the like, and the type thereof does not matter. Further, the recording medium 108 may be fixed to or detachable from the device 100.

The control unit 109 controls each block of the image pickup apparatus 100 by transmitting a control signal to each block of the image pickup apparatus 100 via the bus 114, from a CPU, a memory, or the like for executing various controls. It is composed. The memory used by the control unit 109 is a ROM for storing various control programs, a RAM used as a work area for arithmetic processing, and the like, and also includes an external memory of the control unit 109.

The operation unit 110 is a button, a dial, a touch panel, or a combination thereof, and transmits an instruction signal to the control unit 109 in response to a user operation. Specifically, the operation unit 110 has a shooting button for instructing the start and end of moving image recording, a zoom lever for instructing an optical or electronic zoom operation on the image, and a cross for making various adjustments. It has a key, a decision key, and so on.

The audio output unit 111 outputs the audio data or compressed audio data reproduced by the recording / reproduction unit 107, or the audio data output by the control unit 109 to the speaker 112, the audio terminal, or the like. The external output unit 113 outputs compressed video data, compressed audio data, audio data, etc. reproduced by the recording / playback unit 107 to an external device. The data bus 114 supplies various data such as voice data and image data, and various control signals to each block of the image pickup apparatus 100.

The above is the description of the configuration of the image pickup apparatus 100 in the embodiment. Next, the normal operation of the image pickup apparatus in the embodiment will be described.

In the image pickup apparatus 100 of the present embodiment, power from a power supply unit (not shown) is supplied to each block of the image pickup apparatus in response to an instruction to turn on the power by operating the operation unit 110 by the user. Will be done.

When the power is supplied, the control unit 109 confirms which mode the mode changeover switch of the operation unit 110 specifies, for example, a shooting mode or a reproduction mode, by an instruction signal from the operation unit 110. In the moving image recording mode in the shooting mode, the image data obtained by the imaging unit 101 and the audio data obtained by the audio input unit 102 are stored as one image file. In the reproduction mode, the image file recorded on the recording medium 108 is reproduced by the recording / reproduction unit 107, displayed on the display unit 105, and output from the speaker 112.

In the shooting mode, first, the control unit 109 transmits a control signal to each block of the image pickup apparatus 100 so as to shift to the shooting standby state, and performs the following operations.

The image pickup unit 101 converts the optical image of the subject captured by the photographing optical lens into a moving image signal by the image pickup element, performs analog-digital conversion, image adjustment processing, and the like to generate moving image data. Then, the imaging unit 101 transmits the obtained moving image data to the display processing unit 104 and causes the display unit 105 to display it. The imaging unit 101 outputs a moving image signal in which one frame is horizontal 1920 pixels × vertical 1080 pixels and the frame rate is 30 frames / sec. The user prepares for shooting while looking at the screen displayed in this way.

The voice input unit 102 converts analog voice signals obtained by a plurality of microphones into digital signals, processes the obtained digital voice signals, and generates multi-channel voice data. Then, the obtained voice data is transmitted to the voice output unit 111, and is output as voice from the connected speaker 112 or an earphone (not shown). The user can also adjust the manual volume for determining the recording volume while listening to the sound output in this way.

Next, when the user operates the record button of the operation unit 110 to transmit the shooting start instruction signal to the control unit 109, the control unit 109 transmits the shooting start instruction signal to each block of the imaging device 100. Then, the mode shifts to the moving image recording mode in the shooting mode. The specific processing of the control unit 109 is as follows.

The image pickup unit 101 converts the optical image of the subject captured by the photographing optical lens into a moving image signal by the image pickup element, performs analog-digital conversion, image adjustment processing, and the like to generate moving image data. Then, the obtained moving image data is transmitted to the display processing unit 104 and displayed on the display unit 105. Further, the image pickup unit 101 transmits the obtained image data to the memory 103.

The voice input unit 102 converts analog voice signals obtained by a plurality of microphones into digital signals, processes the obtained digital voice signals, and generates multi-channel voice data. Then, the obtained voice data is transmitted to the memory 103. When there is only one microphone, the obtained analog voice signal is digitally converted to generate voice data, and the voice data is transmitted to the memory 103.

The coding processing unit 106 reads the moving image data and audio data temporarily stored in the memory 103, performs predetermined coding, generates compressed moving image data, compressed audio data, and the like, and stores the compressed moving image data, compressed audio data, and the like in the memory 103 again. do.

The control unit 109 synthesizes the compressed moving image data and the compressed audio data stored in the memory 103, forms a data stream, and outputs the data stream to the recording / reproducing unit 107. When the audio data is not compressed, the control unit 109 synthesizes the audio data stored in the memory 103 and the compressed moving image data, forms a data stream, and outputs the data stream to the recording / reproducing unit 107.

The recording / playback unit 107 writes the data stream as one moving image file to the recording medium 108 under the control of a file system such as UDF or FAT.

The image pickup apparatus 100 will continue the above processing during the moving image recording state. Then, when the user operates the record button of the operation unit 110 to transmit the shooting end instruction signal to the control unit 109, the control unit 109 transmits the shooting end instruction signal to each block of the imaging device 100. , Operate as follows.

The imaging unit 101 and the audio input unit 102 stop generating moving image data and audio data, respectively. The coding processing unit 106 reads out the remaining image data and audio data stored in the memory, performs predetermined coding, and stops the operation when the compressed moving image data, the compressed audio data, and the like are generated. When the audio data is not compressed, the operation is naturally stopped when the generation of the compressed moving image data is completed.

Then, the control unit 109 synthesizes the last compressed moving image data and the compressed audio data or the audio data to form a data stream, and outputs the data stream to the recording / reproducing unit 107.

The recording / playback unit 107 writes the data stream as one moving image file to the recording medium 108 under the control of a file system such as UDF or FAT. Then, when the supply of the data stream is stopped, the moving image file is completed and the recording operation is stopped.

When the recording operation is stopped, the control unit 109 transmits a control signal to each block of the image pickup apparatus 100 so as to shift to the shooting standby state, and returns to the shooting standby state.

Next, the reproduction mode will be described. When the user operates the operation unit 110 to enter the reproduction mode, the control unit 109 transmits a control signal to each block of the image pickup apparatus 100 so as to shift to the reproduction state, and performs the following operations.

The recording / playback unit 107 reads a moving image file composed of the compressed moving image data and the compressed audio data recorded on the recording medium 108, and sends the read compressed moving image data and compressed audio data to the coding processing unit 106.

The coding processing unit 106 decodes the compressed moving image data and the compressed audio data, and transmits each of them to the display control unit 104 and the audio output unit 111. The display control unit 104 causes the display unit 105 to display the decoded moving image data. The audio output unit 111 outputs the decoded audio data to the built-in speaker 112 or an attached external speaker, and reproduces the decoded audio data as sound.

As described above, the image pickup apparatus 100 of the present embodiment can record and reproduce moving images and sounds.

In the present embodiment, when the voice signal is obtained, the voice input unit 102 performs processing such as level adjustment processing of the voice signal obtained by the microphone. This process may always be performed after the device is started, or may be performed after the shooting mode is selected. Alternatively, it may be performed after the mode related to voice recording is selected. Further, in the mode related to voice recording, the above processing may be performed according to the start of voice recording. In the present embodiment, it is assumed that the above processing is performed at the timing when the moving image shooting is started.

FIG. 2 is a block configuration diagram of an imaging unit 101 and an audio input unit 102 of the imaging device 100 of the present embodiment.

The image pickup unit 101 includes an optical lens 201 that captures an optical image of the subject, and an image pickup element 202 that converts the optical image of the subject captured by the optical lens 201 into an electric signal (image signal). Further, the image pickup unit 101 has an image processing section 203 that converts the analog image signal obtained by the image pickup element 202 into a digital image signal, performs image quality adjustment processing to form image data, and transmits the image data to the memory. .. Further, the imaging unit 101 has an optical lens control unit 204 having a known driving mechanism such as a position sensor and a motor for moving the optical lens 201. In the present embodiment, it is described that the optical lens 201 and the optical lens control unit 204 are built in the image pickup unit 101, but the optical lens 201 is an interchangeable lens that can be attached to and detached from the image pickup device 100 via a lens mount. It may be. Further, the optical lens control unit 204 may be provided in the interchangeable lens.

Here, when the user operates the operation unit 110 to input instructions such as zoom operation and focus adjustment, the control unit 109 controls the optical lens 201 to move the optical lens 201 to the optical lens control unit 204 (drive signal). To send. In response to this control signal, the optical lens control unit 204 confirms the position of the optical lens 201 with a position sensor (not shown) and moves the optical lens 201 with a motor or the like (not shown). Further, when the control unit 109 confirms the distance to the image or the subject obtained by the image processing unit 203 and automatically adjusts the distance, the control signal for driving the optical lens is transmitted. Further, when a so-called vibration isolation function for preventing image blurring is provided, the control unit 109 has a control signal for moving the optical lens 201 based on the vibration detected by a vibration sensor (not shown). Will be transmitted to the optical lens control unit 204.

At this time, driving noise due to the movement of the optical lens 201 and driving noise of the motor for moving the optical lens 201 are generated. The optical lens control unit 204 drives the optical lens 201 in response to a control signal for driving the optical lens 201 from the control unit 109. Therefore, the control unit 109 can know (detect or determine) the timing at which the driving noise is generated.

In the present embodiment, by controlling the optical lens 201, for example, zooming of up to 50 times and at least 1 times can be optically performed. This is referred to as an optical zoom in this embodiment. Of course, the magnification of the optical zoom may be greater than or equal to the above or less than or equal to the above. In the optical zoom, the optical lens control unit 204 zooms the optical image of the subject by moving the optical lens of the optical lens 201 in response to an instruction from the control unit 109. Further, the image processing unit 203 has an electronic zoom function that outputs an image signal obtained by zooming in a part of the image signal obtained by the image sensor 202. Further, it has an electronic zoom function that widens the range of the image obtained by the image sensor 202 and outputs an image signal whose image size is zoomed out by the image processing unit 203.

The above is the configuration and operation of the imaging unit 101 in the embodiment. Next, the configuration and operation of the voice input unit 102 will be described.

The image pickup apparatus 100 of the embodiment has two microphones indicated by reference numerals 205a and 205b. These microphones 205a and 205b convert the vibration propagating in the air (medium) into an electric signal and output an audio signal. The microphone 205a is the main (MAIN) microphone, the microphone 205b is the sub (SUB) microphone 205b, and will be hereinafter referred to by this name.

The details will be clarified later, but the main microphone 205a functions as a microphone corresponding to one channel of stereo sound, and mainly acquires sound from outside the sound processing device (in the embodiment, outside the image pickup device 100). It is a microphone to do. Further, the sub microphone 205b is arranged at a position that functions as a microphone corresponding to the other channel of the stereo sound. The sub microphone 205b is a microphone for mainly acquiring the driving noise from the driving unit in the voice processing device (imaging device 100) as compared with the main microphone 205a.

The main microphone 205a outputs an analog audio signal as Mch (main channel), and the sub microphone 205b outputs an analog audio signal as Sch (sub channel). In the present embodiment, the first voice input unit is the main microphone 205a, and the first voice signal is Mch. Further, the second voice input unit is referred to as a sub microphone 205b, and the second voice signal is referred to as Sch. In the present embodiment, since the stereo system is composed of two channels, the arrangement positions of the main microphone 205a and the sub microphone 205b are provided at positions separated by a predetermined distance in the horizontal direction when the imaging unit 101 is held upright. There is. In the embodiment, the number of microphones is set to 2, but a configuration in which more microphones are held may be used.

The analog audio signals obtained by the main microphone 205a and the sub microphone 205b are supplied to the A / D conversion unit 206, where each audio signal is converted into digital audio data. The A / D converter 206 in the present embodiment performs sampling at a sampling rate of 48 KHz and generates 16 bits of digital data per sampling.

The time-series digital audio data of the preset audio signal period (frame) obtained by the A / D transform unit 206 is supplied to the FFT unit 207, where it is fast Fourier transformed and the frequency spectrum for each frequency. Converted to data. In the present embodiment, the frequency spectrum is converted as 1024-point frequency spectrum data from 0 Hz to 48 kHz, and has a frequency spectrum of 512 points up to the Nyquist frequency of 24 kHz. The frequency spectrum data from the main microphone 205a is represented by Main [0] to [511], and the frequency spectrum data from the sub microphone 205b is represented by Sub [0] to [511]. Further, in the present embodiment, the first audio spectrum data is represented by Main [0] to [511], and the second audio spectrum data is represented by Sub [0] to [511]. It is assumed that the subscript of each spectrum data is "0" indicating the lowest frequency and "511" representing the maximum frequency.

The drive sound calculation processing unit 209 determines the subtraction amount of the drive noise for each frequency component of the frequency spectrum data obtained by the FFT unit 207 according to the control signal from the control unit 109 for driving the drive unit. .. This driving noise is generated by driving the optical lens 201. The driving unit in the present embodiment refers to the optical lens 201 driven by the zoom operation and the focus adjustment. The drive sound calculation processing unit 209 outputs NC_Gain [0] to [511] representing the subtraction amount for each frequency spectrum and the drive noise detection signal.

The details will be clarified from the explanation described later, but the sensitivity difference correction unit 208 has the Main [0] to [511] of the current frame according to the drive noise detection signal one frame before from the drive sound calculation processing unit 209. The sensitivity of Sub [0] to [511] is corrected, and the corrected frequency spectrum data Main [0] to [511] and Sub [0] to [511] are output.

The wind noise calculation processing unit 210 detects wind noise from the frequency spectrum data from the FFT unit 207 and determines the subtraction amount. Then, the wind noise calculation processing unit 210 outputs the determined wind noise frequency spectrum data WC_Gain [0] to [511] and the wind noise level signal.

The stereo gain calculation processing unit 211 determines the gains of the stereo Lch (left channel) and Rch (right channel) with respect to the frequency spectrum data from the FFT unit 207. Then, the stereo gain calculation processing unit 211 outputs Gain_L [0] to [511] and Gain_R [0] to [511] representing the gains for each component of the determined frequency spectrum of each channel. Here, the gain of the left channel is Gain_L [0] to [511], and the gain of the right channel is Gain_R [0] to [511].

The total gain calculation unit 212 includes NC_Gain [0] to [511], WC_Gain [0] to [511] determined by the drive sound calculation processing unit 209, the wind noise calculation processing unit 210, and the stereo gain calculation processing unit 211. Gain_L [0] to [511] and Gain_R [0] to [511] are added up, and Total_Gain_L [0] to [511] and Total_Gain_R [0] to [511] are output. Specifically, it is as follows. In the embodiment, the total gain calculation unit 212 functions as a total gain determination unit.
Total_Gain_R [] = NC_Gain [] + WC_Gain [] + Gain_R []
Total_Gain_L [] = NC_Gain [] + WC_Gain [] + Gain_L []

The L / Rch generation unit 213 generates frequency spectra for each frequency of MAIN [0] to [511], Total_Gain_L [0] to [511] determined by the total gain calculation unit 212, and Total_Gain_R [0] to [511]. It is used to generate frequency spectrum data of Lch and Rch (details will be described later). That is, the L / Rch generation unit 213 in this embodiment functions as a stereo generation unit.

The iFFT unit 214 performs an inverse fast Fourier transform on the frequency spectrum data of each channel generated by the L / Rch generation unit 213, and returns the audio signal to the time series of each channel.

The voice processing unit 215 performs processing such as an equalizer. The auto-level controller adjusts the amplitude of the time-series audio signal to a predetermined level (hereinafter, ALC unit 216).

With the above configuration, the voice input unit 102 performs predetermined processing on the voice signal to form voice data, and transmits the voice data to the memory 103.

Next, the recording operation of the image pickup apparatus 100 of the present embodiment will be described with reference to FIG. FIG. 6 is a flowchart showing a recording sequence of the image pickup apparatus 100 of the embodiment.

In S401, the start of recording (REC) is instructed by the operation of the operation unit 110 by the user, so that this process is started. In S402, the control unit 109 connects a voice path for voice recording. After the voice path is established, in S403, the control unit 109 performs initial setting of signal processing including the control described in the present embodiment, and starts the processing. The content of this signal processing will be described later. After that, signal processing including the control described in the present embodiment is performed until the REC sequence is completed.

During the recording processing sequence, the control unit 109 monitors the user's operation on the operation unit 110. Then, when the zoom lever, which is a part of the operation unit 110, is operated by the user, the process proceeds from S404 to S405, and the control unit 109 controls the image pickup unit 101 to perform the zoom process. This zoom process continues until it is determined in S406 that the user has stopped operating the zoom lever. Note that during the zoom process, as described above, driving noise is generated due to the movement of the lens 201, and the noise is superimposed on the ambient sound and recorded.

Then, when the control unit 109 determines that the end of recording is instructed depending on the operation of the operation unit 110 by the user or the situation of the recording medium 108, the control unit 109 proceeds from S407 to S408. At S408, the control unit 109 disconnects the voice path, and then at S409, signal processing also ends.

Next, the details of the audio input unit 102 of the imaging device 100 of the present embodiment will be described with reference to FIG. FIG. 6 is a block diagram showing a detailed configuration of the voice input unit 102 of the present embodiment.

As described above, the voice input unit 102 in the present embodiment has a main microphone 205a and a sub microphone 205b that convert voice vibration propagating in the air into an electric signal and output the voice signal. Further, as described above, the A / D conversion unit 206 samples the analog audio signal at 48 KHz and 16 bits, and converts the analog audio signal into digital audio data.

The sensitivity difference correction unit 208 corrects the sensitivity difference between the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b. Therefore, the sensitivity difference correction unit 208 includes a sensitivity correction integrator 2081, a sensitivity correction detection unit 2082, a correction amount calculation unit 2083, a sensitivity correction gain table 2084, and a sensitivity difference correction gain unit 2085.

The sensitivity correction integrator 2081 changes the level in the time axis direction with respect to the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b. Have a time constant.

The sensitivity correction detection unit 2082 has a level difference "Main [n]" between Main [0] to [511] and Sub [0] to [511], which are frequency spectrum data having a time constant in the sensitivity correction integrator 2081. −Sub [n] ”is calculated for all frequency points. Note that the difference has positive and negative signs.

The correction amount calculation unit 2083 sets Main [n] = Sub [n] when the difference level from the sensitivity correction detection unit 2082 is negative (equivalent to the case of Main [n] <Sub [n]). Therefore, the correction amount of Sub [n] is calculated.

When the difference level from the sensitivity correction detection unit 2082 is positive (equivalent to the case of Main [n] ≥ Sub [n]), it is not necessary to correct Sub [n]. Therefore, in this case, the correction amount calculation unit 2083 outputs 0 as the correction amount of Sub [n].

The sensitivity correction gain table 2084 stores specific correction amounts of each frequency spectrum Sub [0] to [511] calculated by the correction amount calculation unit 2083.

The sensitivity difference correction gain unit 2085 actually executes level correction of each frequency spectrum Sub [0] to [511] based on the sensitivity correction gain table 2084.

Here, the above time constant is set to a unit of several tens of seconds because the purpose is to delay the tracking of the sensitivity correction as much as possible. Further, when the sensitivity correction integrator 2081 receives a drive noise detection signal indicating detection of drive noise by the drive detection unit 2095 described later, the sensitivity correction integrator 2081 stops its operation. This is intended to eliminate the integral during the unstable period in which the optical lens 201 is driven.

The above is a description of each processing unit constituting the sensitivity difference correction unit 208 in the embodiment. Next, the drive sound calculation processing unit 209 will be described.

The drive sound calculation processing unit 209 subtracts the drive noise from the frequency spectrum data from the main microphone 205a and the sub microphone 205b, Main [0] to [511] and Sub [0] to [511], NC_Gain [0] to [511] is determined, and a drive noise detection signal indicating that the drive noise has been detected is output. Therefore, the drive sound calculation processing unit 209 includes a Mch-Sch calculation unit 2091, a drive noise removal gain calculation unit 2092, an hourly amplitude fluctuation detection unit 2093, an hourly phase fluctuation detection unit 2094, a drive detection unit 2095, and an inter-frame amplitude. It has a difference detection unit 2096 and a drive sound subtraction amount integrator 2097.

The Mch-Sch calculation unit 2091 subtracts the frequency spectrum data Sub [0] to [511] from the sub microphone 205b from the frequency spectrum data Main [0] to [511] from the main microphone 205a to obtain the value of the driving noise. Output as a subtraction amount.

However, at the nth point of the frequency spectrum, when Main [n]> Sub [n], the subtraction amount [n] is set to 0. That is, the Mch-Sch calculation unit 2091 outputs a negative value as a subtraction amount [n] on condition that Main [n] -Sub [n] <0 at the nth point of the frequency spectrum.

Further, when Sub [n] is sufficiently large with respect to Main [n] and Main [n] -Sub [n] is lower than a preset threshold value (negative value), the Mch-Sch calculation unit 2091 is driven. A detection signal [n] indicating that noise has been detected is output, and if no, the detection signal is not output. Actually, noise detection may be represented as "1" and non-detection may be represented as "0".

Further, the determination of driving noise detection may be performed by reversing the subtraction relationship and comparing Sub [n] -Main [n] with the threshold value (having a positive value). In this case, the Mch-Sch calculation unit 2091 outputs a signal indicating drive noise detection when the result of this calculation exceeds the threshold value.

The drive detection unit 2095 receives detection signals [0] to [511] for one frame from the Mch-Sch calculation unit 2091, and if one or more detection signals are present in the detection signals [0] to [511], the drive detection unit 2095 is driven in the corresponding frame. Outputs a drive noise detection signal indicating that noise has been detected.

The processing by the Mch-Sch calculation unit 2091 and the drive detection unit 2095 determines whether or not there exists an "i" (i is any of 0 to 511) that satisfies the following equation when the positive threshold value is defined as Th. It can be said that the determination result is output as a signal indicating driving noise detection.
Main [i] + Th <Sub [i]

The hourly amplitude fluctuation detection unit 2093 receives frequency spectrum data Main [0] to [511] from the main microphone 205a and frequency spectrum data Sub [0] to [511] from the sub microphone 205b between frames in the time direction. The amount of amplitude fluctuation of is detected. Specifically, the hourly amplitude fluctuation detection unit 2093 obtains and outputs the difference value between the component value at the nth point of the frequency spectrum of the current frame and the component value at the nth point of the frequency spectrum of the previous frame. .. Then, when the fluctuation amount at the nth point exceeds the preset threshold value, the hourly amplitude fluctuation detection unit 2093 outputs the hourly amplitude fluctuation amount [n], and when it is below the threshold value, outputs 0. do.

The hourly phase fluctuation detection unit 2094 has frequency spectrum data Main [0] to [511] from the main microphone 205a and frequency spectrum data Sub from the sub microphone 205b based on the phase information acquired from the phase difference determination unit 2111 described later. The phase fluctuation amount of [0] to [511] is detected. For example, when the fluctuation amount exceeds a predetermined threshold value at the nth point of the frequency spectrum, the time-by-time phase fluctuation detection unit 2094 outputs the time-by-time phase fluctuation amount [n]. When the fluctuation amount is equal to or less than the threshold value, the time-to-phase fluctuation detection unit 2094 does not output the time-to-phase fluctuation amount [n], or outputs the time-to-phase fluctuation amount [n] = 0.

The frame-to-frame amplitude difference detection unit 2096 determines the amplitude difference between frames in the time direction of Sub [0] to [511], which is frequency spectrum data from the sub microphone 205b, based on the drive noise detection signal from the drive detection unit 2095. Perform detection. For example, when there is a driving noise detection signal at the nth point of the frequency spectrum and the amplitude difference between the previous frame and the current frame exceeds a predetermined threshold value, the frame-to-frame amplitude difference detection unit 2096 performs an inter-frame amplitude difference amount. Output [n]. When the difference is equal to or less than the threshold value, the inter-frame amplitude difference detection unit 2096 does not output the inter-frame amplitude difference amount [n], or outputs the inter-frame amplitude difference amount [n] = 0.

In the same frame, the drive noise removal gain calculation unit 2092 subtracts from the above-mentioned Mch-Sch calculation unit 2095 [0] to [511], and the hourly amplitude fluctuation detection unit 2093 [0]. ~ [511], Hourly Phase Fluctuation Amount [0] to [511] from Hourly Phase Fluctuation Detection Unit 2094, Interframe Amplitude Difference Amount [0] to [511] from Interframe Amplitude Difference Detection Unit 2096, respectively. The result of the above is multiplied by a predetermined system number to calculate the added drive noise removal amounts [0] to [511], and output the result.

The drive noise subtraction amount integrator 2097 gives a time constant to the amount of fluctuation in the time direction with respect to the drive noise removal amount [0] to [511] output from the drive noise removal gain calculation unit 2092, and drives noise removal gain. NC_Gain [0] to [511] (with positive and negative signs) are output.

The above is the configuration and operation of the drive sound calculation processing unit 209 of the embodiment. Next, the wind noise calculation processing unit 210 will be described.

The wind noise calculation processing unit 210 detects wind noise from the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b, and calculates the subtraction amount. The WC_Gain [0] to [511] to be represented and the wind noise level signal are output. The wind noise calculation processing unit 210 includes a wind detection unit 2101, a wind noise gain calculation unit 2102, and a wind noise subtraction amount integrator 2103.

The wind detection unit 2101 has a predetermined number of low frequency regions from the frequency spectra Main [0] to [511] from the main microphone 205a and the frequency spectra Sub [0] to [511] from the sub microphone 205b. The wind noise level is detected according to the correlation of the points. For example, at 10 points in the low frequency range, the wind noise level is obtained and output according to the following equation. The "n" here is 0 to 9 in the case of the embodiment, but this number may be changed as appropriate.
Wind noise level = Σ (Main [n] -Sub [n]) / (Main [n] + Sub [n])
Σ in the above equation indicates the sum of n = 0 to 9.

Further, the wind noise gain calculation unit 2102 has a table having a characteristic line segment as shown in FIG. As shown in the figure, one line segment has a negative gain below a certain frequency and zero gain above that frequency. Then, it includes a plurality of line segments in which the positions of frequencies from which the gain changes from negative to 0 are different from each other. Then, the wind noise gain calculation unit 2102 determines the wind noise gains [0] to [511] using one line segment according to the wind noise level, and outputs the wind noise gains [0] to [511]. In the embodiment, the wind noise gains [0] to [511] are determined by using a table, but the wind noise gains [0] to [511] are determined by using a function having the wind noise level as an argument. May be decided.

The wind noise subtraction integrator 2103 gives a time constant to the fluctuation amount in the time direction with respect to the wind noise gains [0] to [511] output from the wind noise gain calculation unit 2102, and wind noise gain WC_Gain [0]. ] To [511] (with positive and negative signs) are output.

The above is the configuration and operation of the wind noise calculation processing unit 210 in the embodiment. Next, the stereo gain calculation processing unit 211 in the embodiment will be described.

The stereo gain calculation processing unit 211 is based on the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b, and the stereo Lch gain Gain_L [0]. ] To [511] and Rch gains Gain_R [0] to [511] are generated and output. For this purpose, the stereo gain calculation processing unit 211 includes a phase difference determination unit 2111, a stereo gain calculation unit 2112, a stereo suppression unit 2113, a left gain integrator 2114, and a right gain integrator 2115.

The phase difference determination unit 2111 calculates the phase information of Sub [0] to [511] with respect to the frequency spectrum data Main [0] to [511].

For example, when the phase vector of each point in the frequency spectrum data is represented as V (), the phase information [n] of the frequency point n is calculated according to the following equation.
Phase information [n] = | V (Main [n]) × V (Sub [n]) | / (| V (Main [n]) | ・ | V (Sub [n]) |)
Here, "| x |" on the right side represents the absolute value (scalar) of the vector x, "・" in the denominator represents the product of the scalars, and "x" in the molecule represents the outer product which is the sine of the two vectors. There is.

The phase difference determination unit 2111 outputs the phase information [0] to [511] calculated according to the above equation.

The stereo gain calculation unit 2112 calculates the stereo gains [0] to [511] from the phase information [0] to [511] from the phase difference determination unit 2111. For example, at the frequency point n, the gain of each channel is obtained according to the following equation.
Stereo gain for Lch generation = 1 + phase information [n] × emphasis coefficient Stereo gain for Rch generation = 1-phase information [n] × emphasis coefficient The stereo gain calculation unit 2112 is Lch, Rch calculated by the above equation. Outputs the stereo gain [n] of. Here, the emphasis coefficient is changed according to the frequency, and the upper limit is 1 and the lower limit is 0.

The stereo suppression unit 2113 sets the emphasis coefficient to 0 when it receives a detection signal indicating that the drive noise from the Mch-Sch calculation unit 2091 in the drive sound calculation processing unit 209 has been detected. Further, the stereo suppression unit 2113 sets the emphasis coefficient to 0 according to the wind noise level from the wind detection unit 2101 in the wind noise calculation processing unit 210.

The left gain integrator 2114 gives a predetermined time constant to the amount of fluctuation in the time direction with respect to the stereo gains [0] to [511] for Lch generation output from the stereo gain calculation unit 2112, and stereo gains the stereo gains [0] to [511]. It is output as gains Gain L [0] to [511] (with positive and negative signs).

The right gain integrator 2115 gives a predetermined time constant to the amount of fluctuation in the time direction with respect to the stereo gains [0] to [511] for Rch generation output from the stereo gain calculation unit 2112, and stereo gains the stereo gains [0] to [511]. It is output as gains GainR [0] to [511] (with positive and negative signs).

The above is the configuration and operation of the stereo gain calculation processing unit 211 of the embodiment. Next, the total gain calculation unit 212 in the embodiment will be described.

The total gain calculation unit 212 includes NC_Gain [0] to [511], WC_Gain [0] to [511] determined by the drive sound calculation processing unit 209, the wind noise calculation processing unit 210, and the stereo gain calculation processing unit 211. Gain_L [0] to [511] and Gain_R [0] to [511] are added up, and Total_Gain_L [0] to [511] and Total_Gain_R [0] to [511] are output. Specifically, it is the following equation.
Total_Gain_L [] ＝ NC_Gain [] ＋ WC_Gain [] ＋ Gain_L []
Total_Gain_R [] ＝ NC_Gain [] ＋ WC_Gain [] ＋ Gain_R []

Next, the L / Rch generation unit 213 will be described. The L / Rch generation unit 213 uses the total_Gain_L [0] to [511] and the total_Gain_R [0] to [511] determined by the total gain calculation unit 212 from the frequency spectrum data MAIN [0] to [511]. , Create frequency spectrum data for Lch and Rch outputs. The L / Rch generation unit 213 has an Mch / Sch selection unit 2131 and an L / Rch gain addition unit 2132.

The Mch / Sch selection unit 2131 is a range of frequency points of Sub [0] to [511] to be combined with the frequency spectrum data Main [0] to [511] according to the wind noise level by the wind detection unit 2101. Select. Further, the Mch / Sch selection unit 2131 changes the boundary position to be combined from the low frequency point to the high frequency point according to the wind noise level. If the wind is not detected, the Mch / Sch selection unit 2131 does not perform synthesis and outputs the frequency spectrum data Main [0] to [511] as they are.

The L / Rch gain addition unit 2132 receives the frequency spectrum data Main [0] to [511] output from the Mch / Sch selection unit 2132 with respect to the Total_Gain_L [0] to [511] determined by the total gain calculation unit 212. , Total_Gain_R [0] to [511] are used to create frequency spectrum data for the left and right channels (Lch and Rch).

The above is the configuration and operation of the L / Rch generation unit 213 of the embodiment.

The iFFT unit 214 reverse-converts the frequency spectrum data of each channel generated by the L / Rch generation unit 213 (inverse FFT conversion), and returns the original time-series audio signal. The voice processing unit 215 performs processing such as an equalizer. The ALC (auto level controller) 216 adjusts the amplitude of the time-series audio signal to a predetermined level.

With the above configuration, the voice input unit 102 performs predetermined processing on the voice signal to form voice data, transmits the voice data to the memory 103, and stores the voice data.

Here, the mechanical configuration forming a part of the voice input unit 102 of the present embodiment will be described with reference to FIGS. 3 (a) and 3 (b).

FIG. 3A is an external view of the housing of the image pickup apparatus of the present embodiment. With the image pickup device facing the subject, the reference code "a" at a predetermined position on the right side of the photographer is the input hole (opening) of the main microphone 205a, and the reference code "b" at the opposite position on the left side is. It serves as an input hole for the sub microphone 205b. The enlarged view in FIG. 3B is a mechanical component of the main microphone 205a and the sub microphone 205b, which are a part of the voice input unit 102. FIG. 3B is a cross-sectional view showing the mechanical configuration. The exterior portion 102-1 that constitutes the microphone hole, the main microphone bush 102-2a that holds the main microphone 205a, the sub microphone bush 102-2b that holds the sub microphone 205b, and the pressing that presses and holds each microphone bush against the exterior portion. It is composed of a part 103. The exterior portion 102-1 and the pressing portion 103 are made of a molded member such as a PC material, but there is no problem even if it is a metal member such as aluminum or stainless steel. The main microphone bush 102-2a and the sub microphone bush 102-2b are made of a rubber material such as ethylene propylene diene rubber.

Here, the diameter of the microphone hole in the exterior portion will be described. The diameter of the microphone hole (opening area) in the sub microphone 205b is smaller than the diameter (same area) of the microphone hole in the main microphone 205a, and is reduced by a predetermined magnification. The shape of the microphone hole is preferably circular or elliptical, but it may be square. Further, each hole shape may be the same shape or a different shape. It is an object of the above configuration that the driving noise transmitted by air propagating to the microphone inside the image pickup apparatus is less likely to leak from the microphone hole side of the sub microphone 205b to the outside.

Next, the space in front of the microphone composed of the exterior portion 102-1 and the microphone bush will be described. The volume of the space in front of the sub microphone 205b composed of the exterior portion 102-1 and the sub microphone bush 102-2b is the space in front of the main microphone 205a composed of the exterior portion 102-1 and the main microphone bush 102-2a. It is larger than that and has a configuration that secures a volume of a predetermined magnification. The purpose of this configuration is to increase the change in air pressure in the space in front of the sub microphone 205b and emphasize the driving noise.

As described above, the sub microphone 205b input in the mechanical configuration of the microphone input has a configuration in which the amplitude of the driving noise is greatly emphasized with respect to the main microphone 205a input. The relationship between the audio levels of the drive noise input to each microphone is that the main microphone 205a <sub microphone 205b. On the other hand, the level relationship of the sound from outside the device (ambient environment sound, which is the original purpose of collecting sound) input to each microphone by air propagation from the front of the microphone hole is that the main microphone 205a ≥ the sub microphone 205b. Please note.

Here, the operation of the stereo gain calculation processing unit 211 in the audio input unit 102 of the present embodiment will be described with reference to FIGS. 7 to 9.

FIG. 7 shows an example of a sound path from the outside to the microphone built in the image pickup apparatus 100 and a sound path when the built-in optical lens 201 is driven. The microphone at this time corresponds to the main microphone 205a and the sub microphone 205b shown in FIG. As shown in FIG. 7, the distance between the sound source of the ambient sound and the image pickup device 100 is sufficiently larger than the distance between the main microphone 205a and the sub microphone 205b. Therefore, it can be considered that the sound propagation path from the ambient sound source to the main microphone 205a and the sound propagation path from the ambient sound source to the sub microphone 205b are almost the same. However, the optical lens 201 built in the image pickup apparatus is close to the main microphone 205a and the sub microphone 205b. Further, the distance from the motor for moving the optical lens 201 to the microphone may not be uniform, or the sound path in the image pickup apparatus may be different. Therefore, the audio paths (distances) from the optical lens drive system to the main microphone 205a and the sub microphone 205b are significantly different. That is, there is a large difference in the difference between the sound levels of Mch and Sch between the ambient sound and the driving noise. Therefore, there is a large difference between the ambient noise and the driving noise of the optical lens, and these can be easily distinguished.

On the other hand, originally, it is difficult to determine from the left or right side which side the ambient sound is generated from by the magnitude of Mch and Sch. Therefore, the ambient sound can be determined by using the phase of the audio signal. Details will be described.

8 (a) to 8 (c) show the relationship between a certain frequency spectrum data Main [n] and Sub [n].

The stereo gain calculation processing unit 211 is based on the frequency spectrum data Main [0] to [511] from the main microphone 205a and the frequency spectrum data Sub [0] to [511] from the sub microphone 205b, and the stereo Lch gain Gain_L [0]. ] To [511], Rch gain Gain_R [0] to [511] are output. The stereo gain calculation processing unit 211 has the following configuration.

The phase difference determination unit 2111 calculates the phase information of the frequency spectrum data Sub [0] to [511] with respect to the frequency spectrum data Main [0] to [511].

For example, when the ambient sound of the frequency point n is generated from the main microphone 205a side, the relationship between V (Main [n]) and V (Sub [n]) is as shown in FIG. 8A. Even in the microphone arrangement in the present embodiment, the phase does not change even if the magnitude of the frequency spectrum changes. Therefore, the phase information is obtained by using the outer product of V (Main [n]) and V (Sub [n]) (| V (Main [n]) × V (Sub [n]) |).
Phase information [n] = | V (Main [n]) × V (Sub [n]) | / (| V (Main [n]) | ・ | V (Sub [n]) |)
The phase difference determination unit 2111 outputs the phase information [n] calculated by the above equation. The phase information [n] obtained here is, that is, sinθ of V (Main [n]) and V (Sub [n]), and the ambient sound is on the main microphone 205a side (the right side of the user holding the image pickup device 100). ), 0 <phase information [n] ≦ 1.

Further, when the ambient sound of the frequency point n is generated from the sub microphone 205b side, the relationship between V (Main [n]) and V (Sub [n]) is the relationship of the frequency spectrum as shown in FIG. 8 (b). Become. Even in the microphone arrangement in the present embodiment, the phase does not change even if the magnitude of the frequency spectrum changes.

Therefore, the phase information is obtained by using the outer product of V (Main [n]) and V (Sub [n]) (| V (Main [n]) × V (Sub [n]) |).
Phase information [n] = | V (Main [n]) × V (Sub [n]) | / (| V (Main [n]) | ・ | V (Sub [n]) |)
The phase difference determination unit 2111 outputs the phase information [n] calculated by the above equation. The phase information [n] obtained here is, that is, sinθ of V (Main [n]) and V (Sub [n]), and when the ambient sound is from the sub microphone 205b side, 0> phase information [n]. ] ≧ -1.

Further, when the ambient sound of the frequency point n is generated from the same distance as the main microphone 205a and the sub microphone 205b, that is, from the center of the optical lens 201, the relationship between V (Main [n]) and V (Sub [n]) is shown in FIG. The frequency spectrum relationship is as shown in (c). Even in the microphone arrangement in the present embodiment, the phase does not change even if the magnitude of the frequency spectrum changes.

The phase information can be obtained by using the outer product of V (Main [n]) and V (Sub [n]) (| V (Main [n]) × V (Sub [n]) |).
Phase information [n] = | V (Main [n]) × V (Sub [n]) | / (| V (Main [n]) | ・ | V (Sub [n]) |)
The phase difference determination unit 2111 outputs the phase information [n] calculated by the above equation. The phase information [n] obtained here is sinθ of V (Main [n]) and V (Sub [n]), and the ambient sound has phase information [n] ≈0 from the sub microphone 205b side. ..

The stereo gain calculation unit 2112 calculates the stereo gains [0] to [511] using the phase information [0] to [511] determined as described above. For example, at the frequency point n, the stereo gain calculation unit 2112 calculates the gain of each channel according to the following equation.
Stereo gain for Lch generation = 1 + phase information [n] × emphasis coefficient Stereo gain for Rch generation = 1-phase information [n] × emphasis coefficient And the stereo gain calculation unit 2112 is calculated by the above equation. The stereo gain [n] of the channel is output.

FIG. 9 is a diagram showing the emphasis coefficient at each frequency point used in the stereo gain calculation unit 2112.

When the horizontal axis is the frequency point and the vertical axis is the emphasis coefficient, the emphasis coefficient of the frequency you want to emphasize most is set to the maximum value of 1.0, and the emphasis coefficient of the low range where the phase difference is difficult and the high range where the phase difference cannot be determined is The minimum value is 0.

For example, the emphasis coefficient is 1.0 for 1 kHz to 5 kHz to be emphasized most, and 0 for 200 Hz or less.

The emphasis coefficient in the high frequency range where the phase difference cannot be determined is determined by the distance between the main microphone 205a and the sub microphone 205b. For example, if the distance between the main microphone 205a and the sub microphone 205b is 15 mm and the speed of sound is 340 m / s, the correct phase information cannot be obtained at 11.3 kHz or higher, where a half wavelength is inserted between 15 mm, and the left and right sides are reversed. There is a possibility that it will end up. Further, the accuracy is low at 5.7 kHz or higher in which a quarter wavelength is inserted between 15 mm. Therefore, the emphasis coefficient is applied according to the frequency as shown in FIG.

Here, the operations of the drive sound calculation processing unit 209, the total gain calculation unit 212, and the L / Rch generation unit 213 in the voice input unit 102 of the present embodiment will be described with reference to FIGS. 5, 10 to 13.

FIG. 10 shows an example of amplitude spectrum data of each frequency of the main microphone 205a and the sub microphone 205b.

The FFT unit 207 transforms the audio signal of each channel as a frequency spectrum of 1024 points from 0 Hz to 48 kHz. The converted frequency spectrum data shall have a frequency spectrum of 512 points up to the Nyquist frequency of 24 kHz.

As described above with reference to FIGS. 3A and 3B, according to the mechanical configuration of the microphone input of the image pickup apparatus 100 of the embodiment, the sub microphone 205b has an amplitude of driving noise with respect to the main microphone 205a. Generates a highly emphasized signal. That is, in the amplitude spectrum
Ambient sound level: Main microphone 205a ≥ Sub microphone 205b
Drive noise level: Main microphone 205a <Sub microphone 205b
It becomes a relationship with.

FIG. 10 shows an example of the amplitude spectrum data Main [] from the main microphone 205a and the amplitude spectrum data Sub [] from the sub microphone 205b. Further, "Main-Sub" in the figure indicates a subtraction amount [0] to [511] obtained by subtracting Sub [] from Main [], which is calculated by the Mch-Sch calculation unit 2091.

For example, focusing on the amplitude spectrum around the Nth point in Sch, it can be said that Sch> Mch, that is, the driving noise is the dominant point. At this time, in Main-Sub, the subtraction amount exceeding (below) the predetermined zoom detection threshold value is calculated around the Nth point, and the area around the Nth point is detected as an amplitude spectrum regarded as "driving noise". NS. On the other hand, focusing on the amplitude spectrum at the N2 point in Mch, Sch ≦ Mch. In other words, it can be said that the ambient sound is the dominant point. At this time, since the subtraction amount exceeding the zoom detection threshold value is not calculated in Main-Sub, the amplitude spectrum around the N2 point is not detected as the driving noise. The above calculation is executed in the entire range of the amplitude spectra of [0] to [511].

FIG. 11 is a diagram showing an amplitude spectrum of the time series at the frequency N point of the sub microphone 205b.

The illustrated "Sub ch" indicates that the amplitude spectrum data at the Nth point fluctuates in time series.

Sch | t _n −t _(n-1) | indicates the amount of amplitude fluctuation between frames in the time direction calculated by the hourly amplitude fluctuation detection unit 2093 with respect to the amplitude spectrum at the SchN point, and indicates the amount of amplitude fluctuation between frames. It is output as the quantity [n]. For example, focusing on the amplitude spectrum of Sch from t1 to t2, the amount of fluctuation in the time direction is large, and in Sch | t _n −t _(n-1) |, the amount of fluctuation is detected from t1 to t2. The amount of fluctuation over time that exceeds the threshold value is calculated. This operation is executed at all points of the amplitude spectrum of [0] to [511].

12 (a) and 12 (b) are diagrams showing the phase of the time series at the Nth frequency point in the amplitude spectrum from the main microphone 205a and the amplitude spectrum from the sub microphone 205b.

FIG. 3A shows the phase change of the “ambient environment sound” in the time direction by the complex planes Im and Re, and the solid line portion represents Mch and the dotted line portion represents Sch. For t0, t1, t2, t3, and t4, the transition in the time direction is shown.

FIG. 3B shows a change in the phase of “driving noise”.

Here, with respect to the ambient sound, the phases of Mch and Sch are constant in the time transition from t0 to t4. Regarding the driving noise, the phases of Mch and Sch greatly fluctuate in the time transition from t0 to t4. The phase variation in each time direction is detected by the hourly phase variation detection unit 2094, and is output as the hourly phase variation amount [n]. The hourly phase fluctuation detection unit 2094 executes this calculation for all frequency points of the amplitude spectra of [0] to [511].

13 (a) and 13 (b) show an example of the operation timing chart of the Mch-Sch calculation unit 2091.

Main [N], Sub [N], and Main [N] -Sub [N] in the figure (a) are the amplitude spectrum data of Mch at the frequency N point, the amplitude spectrum data of Sch, and Sch from the Mch amplitude spectrum, respectively. The subtraction amount [N] obtained by subtracting the amplitude spectrum is shown. Main [N] -Sub [N] outputs the result of the calculation performed by the Mch-Sch calculation unit 2091.

Here, focusing on the period from t1 to t2 in FIG. 6A, the amplitude spectrum of Sub [N] is much larger than that of Main [N], and the calculation of Main [N] -Sub [N] is performed. The result exceeds the zoom threshold value, is detected as driving noise, and the subtraction amount [N] is output.

Main [N2], Sub [N2], and Main [N2] -Sub [N2] in FIG. 13B are the Mch amplitude spectrum, the Sch amplitude spectrum, and the Mch amplitude spectrum to the Sch amplitude spectrum at the frequency N2 point, respectively. The subtraction amount [n] obtained by subtracting is shown. Here, focusing on the period from t1 to t2 in the figure (b), Main [N2] and Sub [N2] fluctuate at the same level, and the calculation result of Main [N2] -Sub [N2] is also zoomed. There are no results that exceed the threshold. As a result, the driving noise is not detected at the second point of the frequency N2. The Mch-Sch calculation unit 2091 executes the calculation shown in the timing chart in all the amplitude spectra of [0] to [511].

FIG. 5 shows an example of the timing chart of the L / Rch generation unit 213. The zoom drive operation is controlled by the control unit 109, and the optical lens 201 becomes the drive operation at the timings from t1 to t2. The Mch spectrum represents the spectrum at the Nth point of the specific frequency extracted in FIG. Lch and Rch are generated by adding Total_Gain_L and Total_Gain_R determined by the total gain calculation unit 212 to Mch. As shown in the timing chart of the figure, for example, Rch can be emphasized by lowering Total_Gain_L and raising Total_Gain_R with respect to Mch, and it is possible to generate a 2ch stereo signal with 1ch input. be.

Further, even during the driving operation of the optical lens from t1 to t2, it is possible to remove the driving noise with respect to Lch and Rch by lowering Total_Gain_L and Total_Gain_R.

Here, the operation of the sensitivity difference correction unit 208 in the voice input unit 102 of the present embodiment will be described with reference to FIG.

FIG. 14 shows an example of the operation timing chart of the sensitivity difference correction unit 208. In the figure, the zoom detection shows the detection result of the drive noise of the drive detection unit 2095. The input spectrum NPoint indicates the amplitude spectrum of Mch and the amplitude spectrum of Sch at the Nth frequency point. The solid line part shows Mch, and the dotted line part shows Sch.

The input spectrum (integral) NPoint indicates the integration result of Mch and Sch of the sensitivity correction integrator 2081 at the Nth frequency point. The sensitivity adjustment output spectrum NPoint indicates the amplitude spectrum of Mch and the amplitude spectrum of Sch, which are level-corrected by the sensitivity difference correction gain unit 2085 at the Nth frequency point. The solid line part indicates Mch, and the dotted line part indicates Sch.

In FIG. 14, t0 is the timing of starting REC, and from t0 to t1, it represents a sufficiently long time of about several tens of seconds. From the timings t2 to t3, the zoom detection is turned on, and the drive detection unit 2095 indicates that drive noise is generated.

In the input spectrum NPoint, there is a level difference between Mch and Sch at t0 at the start of REC. On the other hand, the input spectrum (integral) NPoint is integrated by the sensitivity correction integrator 2081 and slowly follows the level difference from t0 to t1. The sensitivity adjustment output spectrum NPoint also corrects the integration result of the input spectrum (integral) NPoint by the sensitivity difference correction gain unit 2085 over a sufficient period of time from t0 to t1. This is because the sensitivity difference correction unit 208 aims to correct the sensitivity of the main microphone 205a and the sub microphone 205b, so that it is sufficient to take a sufficient time of about several tens of seconds to correct the level, and transient responsiveness is required. do not.

Further, during the zoom detection ON period from the timing t2 to t3, the sensitivity correction integrator 2081 is stopped. Therefore, due to the generation of driving noise, a large level difference occurs in the amplitude spectrum of Mch and the amplitude spectrum of Sch. However, since the sensitivity correction integrator 2081 is in the stopped state, the value does not follow the level difference. Be retained. As described above, since the sensitivity difference correction unit 208 aims to correct the sensitivity of the main microphone 205a and the sub microphone 205b, it does not need a response to the transient level difference due to the driving noise. The sensitivity difference correction unit 208 executes the correction shown in the timing chart in all the amplitude spectra of [0] to [511].

Here, the operation of the wind noise calculation processing unit 210 in the voice input unit 102 of the present embodiment will be described with reference to FIGS. 15 to 17.

FIG. 15 is a cross-sectional view showing a mechanical configuration in which the windshield material 102-3 is configured with respect to the sub microphone 205b which is a part of the voice input unit 102.

The exterior portion 102-1 constituting the microphone hole presses and holds the main microphone bush 102-2a for holding the main microphone 205a, the sub microphone bush 102-2b for holding the sub microphone 205b, and the respective microphone bushes against the exterior portion. It is composed of a pressing portion 103. The exterior portion 102-1 and the pressing portion 103 are made of a molded member such as a PC material, but there is no problem even if it is a metal member such as aluminum or stainless steel. The main microphone bush 102-2a and the sub microphone bush 102-2b are made of a rubber material such as ethylene propylene diene rubber.

Here, the hole diameter of the microphone hole in the exterior portion 102-1 will be described. The diameter of the microphone hole in the sub microphone 205b is smaller than the diameter of the microphone hole in the main microphone 205a. In the embodiment, the diameter (diameter) of the microphone hole of the sub microphone 205b is set to 1/3 of the size of the microphone hole of the main microphone 205a. The shape of the microphone hole is preferably circular or elliptical, but it may be square. Further, each hole shape may be the same shape or a different shape.

Next, the space in front of the microphone composed of the exterior portion 102-1 and the microphone bushes 102-2a and 102-2b, and the arrangement of the cushion material will be described. The volume of the space in front of the sub microphone 205b composed of the exterior portion 102-1 and the sub microphone bush 102-2b is that of the space in front of the main microphone 205a composed of the exterior portion 102-1 and the main microphone bush 102-2a. It is configured to be larger and secure three times.

In the space in front of the sub microphone 205b composed of the exterior portion 102-1 and the sub microphone bush 102-2b, a windshield cushion material and a seal microphone are arranged as the windshield material 102-3. Each is configured as a member for filtering a signal component in a low frequency band of about 0 to 4 kHz corresponding to the wind frequency. The windshield material 102-3 can significantly reduce the influence of air propagation on the sub microphone 205b of wind noise, which is dominated by the low frequency band.

FIG. 16 shows frequency spectrum data Main [0] to [511] from the main microphone 205a and frequency spectrum data Sub [0] to [511] from the sub microphone 205b at the time of wind noise input. At the time of wind noise input, the wind noise component exists in the low frequency band of the dotted line portion. The wind detection unit 2101 correlates, for example, 10 points in the low frequency band from the frequency spectra Main [0] to [511] from the main microphone 205a and the frequency spectra Sub [0] to [511] from the sub microphone 205b. The wind noise level is detected by looking at. The wind detection unit 2101 calculates and outputs the wind noise level according to the following equation, for example, at the low frequency point n.
Wind noise level = Σ (Main [n] -Sub [n]) / (Main [n] + Sub [n])
In the above equation, 10 points of the low frequency component are set, and n is in the range of 0 to 9. Further, in the embodiment, the low frequency band is set to 10 points, but this number is an example. It is desirable to make appropriate settings according to the design of the imaging device.

FIG. 17 shows the frequency relationship of the wind noise gains [0] to [511] with respect to the wind noise level from the wind detection unit 2101, which is calculated by the wind noise gain calculation unit 2102. The larger the wind noise level from the wind detection unit 2101, the more the wind noise gain shifts to the minus side, and the cutoff frequency indicated by the dotted line shifts to the high frequency band. The wind noise gains [0] to [511] are determined by the cut-off frequency.

Next, the operation of the Mch / Sch selection unit 213 in the voice input unit 102 of the present embodiment will be described with reference to FIGS. 18A and 18B.

FIG. 18A shows frequency spectrum data Main [0] to [511] (shown Main ch) from the main microphone 205a and frequency spectrum data Sub from the sub microphone 205b, which are synthesized by the Mch / Sch selection unit 2131. The relationship between the combined ratio and the frequency according to the wind noise level between [0] to [511] (Subch in the figure) is shown.

Here, FIG. 3A shows an embodiment corresponding to the mechanical configuration of the main microphone 205a and the sub-microphone 205b shown in FIG. 3B. Here, as shown in FIG. 18A, the Mch / Sch selection unit 2131 sets the Main ch at a ratio of 1.0 to 0.5 and the Sub ch at a ratio of 0 to 0.5 based on the wind noise level. Synthesize at the ratio of.

The higher the wind noise level, the lower the synthesis ratio from 1.0 to 0.5 in Main ch, and the higher the synthesis ratio from 0 to 0.5 in Sub ch, and the crossover that synthesizes Main ch and Sub ch. Increase the frequency (upper limit frequency of synthesis). Then, the Mch / Sch selection unit 2131 synthesizes the Main ch and the Sub ch at the ratio shown in the figure below the upper limit frequency depending on the wind noise level, and selects and outputs the Main ch at the frequency exceeding the upper limit frequency. When the wind noise level is 0, the composition ratio of Sch is 0. Here, as described in FIG. 3B, the diameter of the microphone hole in the sub microphone 205b is smaller than the diameter of the microphone hole in the main microphone 205a, and is reduced to 1/3. Therefore, the influence of wind noise on the sub microphone 205b is weaker than that of the main microphone 205a. Therefore, by synthesizing Sch with Mch according to the wind noise level from the wind detection unit 2101, the wind noise can be reduced effectively.

Next, FIG. 18B shows an embodiment corresponding to the mechanical configuration in which the windshield material 102-3 is configured with respect to the sub microphone 205b as shown in FIG. Here, the Mch / Sch selection unit 2131 synthesizes Mch at a ratio of 1.0 to 0 and Sch at a ratio of 0 to 1.0 based on the wind noise level. That is, the larger the wind noise level, the lower the synthesis ratio from 1.0 to 0 in Mch, the higher the synthesis ratio from 0 to 1.0 in Sch, and the higher the frequency of the crossover that synthesizes Mch and Sch. go. When the wind noise level is 0, the composition ratio of Sch is 0. Here, as described in FIGS. 3A and 15, the diameter of the microphone hole in the sub microphone 205b is smaller than the diameter of the microphone hole in the main microphone 205a, and is reduced to 1/3. In addition, a windshield material 102-3 is provided in the space in front of the sub microphone 205b composed of the exterior portion 102-1 and the sub microphone bush 102-2b. Therefore, the degree of influence of wind noise on the sub microphone 205b can be made smaller than that of the main microphone 205a. Therefore, by switching from Mch to Sch according to the wind noise level from the wind detection unit 2101, the wind noise can be reduced effectively.

Here, the specific operation of the stereo suppression unit 2113 in the voice input unit 102 of the present embodiment will be described with reference to FIGS. 19 and 20.

FIG. 19 shows a timing chart of the stereo suppression unit 2113 for changing the emphasis coefficient used for enhancing the stereo effect according to the drive noise detection and the wind noise detection. In FIG. 19, Main [N] indicates the amplitude spectrum data of Mch at the frequency N point. Further, the drive noise detection signal indicates a detection signal indicating that the drive noise has been detected by the drive detection unit 2095. Further, the wind noise detection signal indicates a wind noise level (a wind noise level equal to or higher than a preset threshold value) indicating that the wind noise has been detected by the wind detection unit 2101. GainL [N] and GainR [N] indicate stereo Lch and Rch gain to be added to the amplitude spectrum of Mch at the Nth frequency point determined by the stereo gain calculation processing unit 2112.

The stereo suppression unit 2113 receives a detection signal indicating that the drive noise from the Mch-Sch calculation unit 2091 has been detected, and sets the emphasis coefficient to 0. Further, the emphasis coefficient is set to 0 according to the frequency in response to the wind noise level indicating that the wind noise from the wind detection unit 2101 has been detected.

Here, focusing on the period from timing t1 to t2, the amplitude spectrum of Main [N] fluctuates greatly, and the detection signal from the Mch-Sch calculation unit 2091 indicates detection as having driving noise. During this period, GainL [N] and GainR [N] are fixed at 0. That is, it means that the stereo suppression unit 2113 has set the emphasis coefficient to 0. Further, focusing on the period from timing t3 to t4, the amplitude spectrum of Main [N] fluctuates greatly, and the wind noise detection signal from the wind detection unit 2101 indicates detection. During this period, GainL [N] and GainR [N] are fixed at 0. That is, it means that the stereo suppression unit 2113 has set the emphasis coefficient to 0.

FIG. 20 shows the frequency spectra Main [0] to [511] from the main microphone 205a and the frequency spectra Sub [0] from the sub microphone 205b by the Mch / Sch selection unit 2131 when the wind noise level is detected from the wind detection unit 2101. ] To [511] are combined, and the relationship between the frequency and the frequency at which the emphasis coefficient is set to 0 by the stereo suppression unit 2113 is shown. Here, the Mch / Sch selection unit 2131 reduces the composition ratio from 1.0 to 0.5 in Mch and increases the composition ratio from 0 to 0.5 in Sch as the wind noise level increases from the wind noise level. , The frequency of the crossover that synthesizes Mch and Sch is increased. In the case of wind noise level, the frequency of the crossover is 500 Hz. On the other hand, the stereo suppression unit 2113 fixes the emphasis coefficient to 0 up to a frequency of 750 Hz, which is higher than the crossover frequency. The stereo suppression unit 2113 raises the frequency at which the emphasis coefficient is fixed to 0 as the wind noise level from the wind detection unit 2101 increases. The enhancement by the stereo gain of GainL and GainR prevents the wind noise from being emphasized.

Here, the operations of the drive sound subtraction integrator 2097, the wind noise subtraction integrator 2103, the right gain integrator 2114, and the left gain integrator 2115 in the audio input unit 102 of the present embodiment will be described with reference to FIG. ..

FIG. 21 shows the drive noise removal gain NC_GAIN [N], the wind noise subtraction amount WC_GAIN [N], the stereo gain L_GAIN [N] for Lch generation, and Rch, respectively, for the amplitude spectrum data of the Mch at the Nth frequency point. The time constants for each of the generation stereo gains R_GAIN [N] are shown. These are determined by the drive sound subtraction integrator 2097, the wind noise subtraction integrator 2103, the left gain integrator 2114, and the right gain integrator 2115. The time constant of the drive noise subtraction integrator is slower than the time constant of the right gain integrator 2115 and the left gain integrator 2114, and the time constant of the wind noise subtraction integrator is the right gain integrator 2115 and the left gain integrator. Slow with respect to the time constant of 2114. The drive noise and the wind noise are drive noise components, respectively, and the variation in the time series is large. The variation is suppressed by delaying the time constant and delaying the follow-up of the drive noise subtraction and the wind noise subtraction, respectively. As for the stereo gain, by increasing the time constant, the tracking of the movement of the sounding subject is accelerated.

In the present embodiment, the case where two systems of audio are input has been described, but it can be applied even if the number of channels is larger than that.

Further, in the present embodiment, the imaging device has been described, but what if the voice processing of the voice input unit 102 of the present embodiment is a device that records or inputs an external voice, that is, a voice recording device? It can be applied to any device. For example, it may be applied to an IC recorder, a mobile phone, or the like.

Further, in the embodiment, an example in which the configuration shown in FIG. 6 is realized by hardware has been described. It may be realized by the program of.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 ... Imaging device, 101 ... Imaging unit, 102 ... Audio input unit, 103 ... Memory, 104 ... Display control unit, 105 ... Display unit, 106 ... Coding processing unit, 107 ... Recording / playback unit, 108 ... Recording medium, 109 ... Control unit, 110 ... Operation unit, 111 ... Sound output unit, 112 ... Speaker, 113 ... External output unit, 114 ... Data bus, 201 ... Optical lens, 202 ... Imaging element, 203 ... Image processing unit, 204 ... Optical lens Control unit, 205 ... Microphone, 205a ... Main microphone, 205b ... Sub microphone, 206 ... A / D conversion unit, 207 ... FFT unit, 208 ... Sensitivity difference correction unit, 209 ... Drive sound calculation processing unit, 210 ... Wind noise calculation processing Unit, 211 ... Stereo gain calculation processing unit, 212 ... Total gain calculation unit, 213 ... L / Rch generation unit, 214 ... iFFT unit, 215 ... Sound processing unit, 216 ... ALC unit, 102-1 ... Exterior unit, 102- 2a ... Main microphone bush, 102-2b ... Sub microphone bush, 102-3 ... Windshield material

Claims (11)

It is a voice processing device
With the first microphone
With the second microphone
Conversion that converts the time-series audio data obtained from the first microphone into the first frequency spectrum data and converts the time-series audio data obtained from the second microphone into the second frequency spectrum data. Means and
As a determination means for obtaining the phase shift of the second frequency spectrum data with respect to the first frequency spectrum data obtained by the conversion means and determining the gain of the right channel and the gain of the left channel based on the phase shift. ,
The first frequency spectrum data is controlled by the gain of the right channel determined by the determination means to generate the frequency spectrum data of the right channel, and the first frequency spectrum data is determined by the determination means. A generation means that is controlled by the gain of the left channel to generate frequency spectrum data of the left channel,
Have a reverse conversion means for inversely converting the respective frequency spectrum data of the left and right channels generated by said generating means, to each of the audio data of the left and right channels of the time series,
The voice processing device is characterized in that, when a predetermined noise is detected, the determination means determines the gain of the right channel and the gain of the left channel without being based on the phase shift. When the first microphone is located on the right side and the second microphone is located on the left side of the object to be collected.
The determination means is
A means for acquiring phase information of the second frequency spectrum data with respect to the first frequency spectrum data, and
A means for determining the gain of the right channel and the gain of the left channel based on the calculated phase information and the emphasis coefficient corresponding to the value obtained by subtracting the second frequency spectrum data from the first frequency spectrum data. The voice processing apparatus according to claim 1, wherein the voice processing apparatus includes. The determination means is
When the phase of the first frequency spectrum data is ahead of the phase of the second frequency spectrum data, the gain of the left channel is increased.
The voice processing apparatus according to claim 2, wherein when the phase of the first frequency spectrum data is behind the phase of the second frequency spectrum data, the gain of the right channel is increased. The determination means is characterized in that the gain of the right channel and the gain of the left channel are determined by multiplying the sine of the second frequency spectrum data with respect to the first frequency spectrum data by an emphasis coefficient. 2. The voice processing device according to 2. The voice processing apparatus according to claim 2, wherein the emphasis coefficient is obtained for each frequency. The voice processing device according to claim 2, wherein the determination means determines the emphasis coefficient based on the distance between the first microphone and the second microphone. It also has a drive unit
The voice processing device according to any one of claims 1 to 6, wherein the predetermined noise is driving noise by the driving unit. The voice processing device according to any one of claims 1 to 6, wherein the predetermined noise is wind noise. The determination means is any one of claims 1 to 8, wherein when a predetermined noise is detected, the gains of the right channel and the gains of the left channel are determined to be the same. The audio processing device described in the section. It is a control method of the voice processing device.
The voice processing device has a first microphone and a second microphone.
The method is
Conversion that converts the time-series audio data obtained from the first microphone into the first frequency spectrum data and converts the time-series audio data obtained from the second microphone into the second frequency spectrum data. Process and
A determination step of obtaining the phase shift of the second frequency spectrum data with respect to the first frequency spectrum data obtained in the conversion step and determining the gain of the right channel and the gain of the left channel based on the phase shift. ,
The first frequency spectrum data is controlled by the gain of the right channel determined by the determination step to generate the frequency spectrum data of the right channel, and the first frequency spectrum data is determined by the determination step. A generation process that generates frequency spectrum data for the left channel by controlling it with the gain of the left channel,
Have a reverse transformation step of inverse transforming each of the frequency spectrum data of the left and right channels generated in the generation step, each of the audio data of the left and right channels of the time series,
The determination step is a control method of a voice processing apparatus, characterized in that, when a predetermined noise is detected, the gain of the right channel and the gain of the left channel are determined without being based on the phase shift. A program that is read and executed by a computer having a first microphone and a second microphone that function as stereo microphones for sound from outside the device.
A program for causing the computer to execute each step of the method according to claim 10.

Images