JP5063489B2 - Judgment device, electronic apparatus including the same, and judgment method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a determination device capable of accurately carrying out direction determination for a broadband sound signal, even if intervals between a plurality of microphones are small. <P>SOLUTION: This determination device includes: an FFT part 23 for subjecting an output signal of a first microphone to time-frequency conversion; an FFT part 24 for subjecting an output signal of a second microphone having a directional characteristic different from that of the first microphone to time-frequency conversion; and a power comparison spectrum determination part 25 having a power comparison part for comparing power of a signal S1[F], in a frequency region output from the FFT part 23, with power of a signal S2[F] in a frequency region output from the FFT part 24, on frequency basis in a predetermined frequency band, and a determination part for determining a sound or sound source direction from a specific direction by using the comparison result in the power comparison part. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to a determination device that determines a sound or sound source direction from a specific direction, an electronic apparatus including the determination device, and a determination method that determines a sound or sound source direction from a specific direction.

  As a conventional specific sound source enhancement technique, there is one disclosed in Patent Document 1, for example. The speech processing unit for realizing the conventional specific sound source enhancement method disclosed in Patent Document 1 has a configuration as shown in FIG. In the audio processing unit shown in FIG. 14, an FFT (Fast Fourier Transform) unit 101 converts the output signal of the first microphone into a digital signal, and then converts it into a frequency domain signal, and the FFT unit 102 uses the second microphone. Is converted to a digital signal and then further converted to a frequency domain signal, and the spectrum determination unit 103 calculates from the frequency domain signal output from the FFT unit 101 and the frequency domain signal output from the FFT unit 102. The required spectrum is determined based on the relative parameters, and the unnecessary spectrum units 104 and 105 are controlled based on the determination result. The unnecessary spectrum unit 104 determines the unnecessary spectrum of the frequency domain signal output from the FFT unit 101. Attenuate the unnecessary spectrum portion 105 of the frequency domain signal output from the FFT portion 102. The unwanted spectrum is attenuated, the frequency domain signal output from the unwanted spectrum section 104 is converted into time-series data by an IFFT (Inverse Fast Fourier Transform) section 106, and the frequency domain signal output from the unwanted spectrum section 105 is converted to IFFT. The unit 107 converts the data into time series data.

Japanese Patent No. 3435357

  When phase information is used as a relative parameter, the controllable upper limit frequency is determined according to the interval between the first microphone and the second microphone. The upper limit frequency that can be controlled becomes higher as the first microphone and the second microphone are brought closer to each other. However, considering the size of a general microphone, the center distance between the first microphone and the second microphone is 2 cm. The upper limit frequency is around 8 kHz.

  On the other hand, when power information is used as a relative parameter, there is no restriction on the lower limit / upper limit frequency, but when sound comes from a specific direction with respect to the first microphone 108 and the second microphone 109 as shown in FIG. When the distance between the first microphone 108 and the second microphone 109 is short, the difference d between the path from the sound source to the first microphone 108 and the path from the sound source to the second microphone 109 becomes small. Since the attenuation of the sound corresponding to the path difference d becomes minute, it is difficult to identify the relative parameter.

  Therefore, the conventional specific sound source enhancement method disclosed in Patent Document 1 cannot accurately determine the direction of a wideband audio signal when the distance between the first microphone and the second microphone is short. .

  In view of the above situation, the present invention provides a determination device capable of accurately performing direction determination on a wideband audio signal even when the interval between a plurality of microphones is short, an electronic device including the determination device, and a determination method. Objective.

  In order to achieve the above object, a determination apparatus according to the present invention includes a first time-frequency conversion unit that performs time-frequency conversion on an output signal of a first microphone, and a second directional characteristic different from that of the first microphone. A second time-frequency converter that converts the output signal of the microphone to time-frequency, a power of a frequency-domain signal output from the first time-frequency converter, and a frequency output from the second time-frequency converter. A power comparison unit that compares the signal power of the region for each frequency in a predetermined frequency band; and a determination unit that determines a sound or sound source direction from a specific direction using a comparison result in the power comparison unit. The configuration. The power at a certain frequency of the frequency domain signal can be expressed by, for example, the square root of the square of the amplitude at the certain frequency of the frequency domain signal.

  According to such a configuration, it is a comparison result between the power of the frequency domain signal output from the first time frequency converter and the power of the frequency domain signal output from the second time frequency converter. Since the direction determination is performed using the relative power parameter, there is no restriction on the lower limit and the upper limit frequency, and the direction determination for the wideband audio signal can be performed with high accuracy. Further, since the directivity characteristics of the first microphone and the second microphone are different from each other, the change in the relative power parameter due to the difference in the sound source direction can be increased. Therefore, even when the interval between the first microphone and the second microphone is short, the direction determination for the audio signal can be performed with high accuracy.

  A storage unit that stores a determination condition based on a difference in directivity between the first microphone and the second microphone; and the determination unit stores the comparison result in the power comparison unit and the storage unit. You may make it determine the sound from a specific direction from the stored determination conditions.

  A third time-frequency conversion unit that performs time-frequency conversion on an output signal of a third microphone that has the same frequency direction as the second microphone, and the predetermined frequency band is a first frequency band; The phase of the frequency domain signal output from the second time frequency converter and the phase of the frequency domain signal output from the third time frequency converter in a band lower than the first frequency band. A phase comparison unit for comparing each frequency in a certain second frequency band, and a first storage unit for storing a first determination condition based on the amount of difference in directivity between the first microphone and the second microphone And a second storage unit that stores a second determination condition based on a positional relationship between the second microphone and the third microphone, wherein the determination unit is a comparison result of the power comparison unit The sound from the specific direction of the first frequency band is determined based on the first determination condition stored in the first storage unit, and the comparison result in the phase comparison unit and the second storage unit You may make it determine the sound from the specific direction of the said 2nd frequency band from the stored 2nd determination conditions.

  Further, the phase of the frequency domain signal output from the first time-frequency converter and the frequency domain signal output from the second time-frequency converter so that sound in two directions can be determined. And a first comparison condition that stores a first determination condition based on a difference in directivity characteristics between the first microphone and the second microphone. 1 storage unit, and a second storage unit that stores a second determination condition based on the positional relationship between the first microphone and the second microphone, and the determination unit is the power comparison unit Primary determination to determine whether the sound is from the first direction or the sound from the second direction from the comparison result of the first and the first determination condition stored in the first storage unit And the primary determination unit When it is determined that the sound is from the first direction or the sound from the second direction, the comparison result in the phase comparison unit and the second stored in the second storage unit And a secondary determination unit that determines whether or not the sound is from the first direction.

  Further, the directivity pattern of the first microphone and the directivity pattern of the second microphone are symmetric, and the power comparison unit outputs a frequency domain output from the first time frequency conversion unit. When a comparison result is obtained that the power of the signal is equal to the power of the signal in the frequency domain output from the second time-frequency conversion unit, the determination unit determines that the sound is from the front direction. You may make it do.

  In order to achieve the above object, an electronic apparatus according to the present invention includes at least one determination device having the above-described configuration, and performs sound processing on the collected sound signal based on the determination result of the determination device. .

  Further, the electronic device having the above-described configuration has a recording / playback function of the collected audio signal, and the determination device is used either when recording the collected audio signal or reproducing the recorded audio signal May perform the determination process.

  Moreover, as an example of the electronic device having each configuration described above, an imaging apparatus including a camera that captures an image can be given.

  In order to achieve the above object, a determination method according to the present invention includes a first time-frequency conversion step of converting the output signal of the first microphone to time-frequency, and a second directional characteristic different from that of the first microphone. A second time-frequency conversion step for time-frequency conversion of the output signal of the microphone; a power of the signal in the frequency domain obtained by the first time-frequency conversion step; and a frequency domain obtained by the second time-frequency conversion step. A power comparison step of comparing the power of the signal for each frequency in a predetermined frequency band; and a determination step of determining a sound or sound source direction from a specific direction using a comparison result obtained by the power comparison step.

  According to the present invention, since the direction determination is performed using the relative power parameter, there is no restriction on the lower limit and the upper limit frequency, and the direction determination with respect to the wideband audio signal can be performed with high accuracy. In addition, since the output signals of a plurality of different microphones are used, the change in relative power parameter due to the difference in sound source direction can be increased. Therefore, even when the interval between the plurality of microphones is short, the direction determination for the audio signal can be performed with high accuracy.

  Embodiments of the present invention will be described below with reference to the drawings.

  Since the determination method according to the present invention makes a determination using the collected audio signal, not only when the collected audio signal is recorded, but also when the already collected / recorded audio signal is reproduced. Applicable.

  Hereinafter, an image pickup apparatus (for example, a video camera, a digital camera, etc.) equipped with a determination apparatus applied when recording an audio signal collected by the determination method according to the present invention will be described as an example.

  FIG. 1 is a block diagram showing an example of an internal configuration of an imaging apparatus equipped with a determination apparatus according to the present invention.

  The image pickup apparatus shown in FIG. 1 has a solid-state image pickup device (image sensor) 1 such as a CCD (Charge Coupled Device) or CMOS (Complimentary Metal Oxide Semiconductor) sensor that converts incident light into an electric signal, and an optical image of a subject. The image sensor 1 outputs a zoom lens that forms an image on the image sensor 1, a motor that changes the focal length of the zoom lens, that is, a motor that changes the optical zoom magnification, and a motor that focuses the zoom lens on the subject. An AFE (Analog Front End) 3 that converts an image signal that is an analog signal into a digital signal, a microphone unit 4 that has a plurality of microphones having different directivity characteristics, and an image signal that is a digital signal from the AFE 3 An image processing unit 5 that performs various image processing such as correction, and an audio signal that is an analog signal from the microphone unit 4 On the other hand, the audio processing unit 6 that converts the digital signal and performs audio correction processing, and the MPEG (Moving Picture Experts Group) compression method for the image signal from the image processing unit 5 and the audio signal from the audio processing unit 6, etc. A compression processing unit 7 that performs the compression encoding process, a driver unit 8 that records the compression-encoded signal compression-encoded by the compression processing unit 7 in an external memory 22 such as an SD card, and the driver unit 8 an external memory 22 A decompression processing unit 9 that decompresses and decodes the compression-coded signal read from the video signal, a video output circuit unit 10 that converts an image signal obtained by decoding by the decompression processing unit 9 into an analog signal, and a video output circuit unit 10 A video output terminal 11 for outputting the signal converted in step S4, a display unit 12 having an LCD or the like for displaying an image based on the signal from the video output circuit unit 10, and decompression. Based on the audio output circuit unit 13 that converts the audio signal from the processing unit 9 into an analog signal, the audio output terminal 14 that outputs the signal converted by the audio output circuit unit 13, and the audio signal from the audio output circuit unit 13 A speaker unit 15 that reproduces and outputs sound, a timing generator (TG) 16 that outputs a timing control signal for matching the operation timing of each block, and a CPU (Central Processing Unit) that controls the overall driving operation in the imaging apparatus. ) 17, a memory 18 for storing each program for each operation and temporarily storing data when the program is executed, an operation unit 19 to which an instruction from the user is input, and between the CPU 17 and each block A bus line 20 for exchanging data and a bus circuit for exchanging data between the memory 18 and each block. It includes a 21, a. The CPU 17 controls the focus and the diaphragm by driving each motor in the lens unit 2 in accordance with the image signal detected by the image processing unit 5.

  The audio processing unit 6 includes the determination device according to the present invention, and performs audio processing according to the determination result of the determination device according to the present invention. From the viewpoint of downsizing and cost reduction, the audio processing unit 6 alone or the audio processing unit 6 including the audio compression encoder in the compression processing unit 7 is an LSI package that is made into one package. Is desirable.

  Next, the basic operation of the image pickup apparatus shown in FIG. 1 during moving image shooting will be described with reference to the flowchart of FIG. First, when the user operates the operation unit 19 to set the imaging device for moving image shooting and turn on the power (STEP 1), the driving mode of the imaging device, that is, the driving mode of the image sensor 1 is set to the preview mode ( (Step 2). Subsequently, the camera enters a shooting mode input waiting state. When the shooting mode is not input, it is assumed that the normal shooting mode is selected (STEP 3). In the preview mode, an image signal that is an analog signal obtained by the photoelectric conversion operation of the image sensor 1 is converted into a digital signal by the AFE 3, subjected to image processing by the image processing unit 5, and compressed by the compression processing unit 7. An image signal for the current image is temporarily recorded in the external memory 22. The compressed signal is expanded by the expansion processing unit 9 via the driver unit 8, and an image with an angle of view at the zoom magnification of the lens unit 2 set at the present time is displayed on the display unit 12.

  Subsequently, when the user operates the operation unit 19 so as to obtain a desired angle of view with respect to the subject to be photographed, the zoom magnification with the optical zoom corresponding to the operation is set (STEP 4). At this time, the lens unit 2 is controlled by the CPU 17 based on the image signal input to the image processing unit 5, and optimum exposure control (Automatic Exposure; AE) / focusing control (Auto Focus; AF) is performed. It is performed (STEP 5).

  Thereafter, when the recording start button of the operation unit 19 (which may also be used as a shutter button for taking a still image) is fully pressed and the start of the recording operation is instructed (Y in STEP 6), the recording operation is started and the image is displayed. An image signal that is an analog signal obtained by the photoelectric conversion operation of the sensor 1 is output to the AFE 3. At this time, the image sensor 1 receives the timing control signal from the TG 16 to perform horizontal scanning and vertical scanning, and output an image signal as data for each pixel. Then, in the AFE 3, the image signal (raw data) that is an analog signal is converted into a digital signal and written into the frame memory in the image processing unit 5 (STEP 7).

  The image processing unit 5 performs various image processing such as signal conversion processing for generating a luminance signal and a color difference signal, and the image signal subjected to the image processing is given to the compression processing unit 7. On the other hand, the audio processing unit 6 performs A / D conversion processing on an audio signal that is an analog signal obtained by inputting the sound into the microphone unit 4, and the determination result of the determination device according to the present invention. The audio signal is subjected to the audio processing and the audio signal subjected to the audio processing is supplied to the compression processing unit 7 (STEP 8). This voice processing will be described later.

  The compression processing unit 7 compresses and encodes the image signal and the audio signal, which are digital signals, based on the MPEG compression coding method (STEP 9), gives them to the driver unit 8, and records them in the external memory 22 (STEP 10). . At this time, the compressed data recorded in the external memory 22 is read out by the driver unit 8 and given to the expansion processing unit 9, and is subjected to expansion processing to obtain an image signal. This image signal is given to the display unit 12 to display a subject image currently photographed through the image sensor 1. After that, when the recording start button of the operation unit 19 is fully pressed again and the end of the recording operation is instructed (Y in STEP 11), the process returns to the preview mode (STEP 2).

  When performing such an imaging operation, a timing control signal is given by the TG 16 to the AFE 3, the image processing unit 5, the audio processing unit 6, the compression processing unit 7, and the decompression processing unit 9, and one frame by the image sensor 1. An operation synchronized with each imaging operation is performed.

  In addition, when it is instructed through the operation unit 19 to reproduce the compressed moving image data recorded in the external memory 22, the compressed moving image data recorded in the external memory 22 is read by the driver unit 8 and decompressed. 9 is given. Then, the decompression processing unit 9 decompresses and decodes the image signal and the audio signal based on the MPEG compression encoding method. Then, the image signal is given to the display unit 12 to reproduce the image, and the audio signal is given to the speaker unit 15 via the audio output circuit unit 13 to reproduce the audio. Thereby, an image based on the compressed moving image data recorded in the external memory 22 is reproduced together with the sound.

  Next, four embodiments (first to fourth embodiments) are shown as specific examples of the audio processing unit 6, and the audio processing performed by the audio processing unit 6 of each embodiment in STEP 8 of FIG. 2 will be described. .

<First Embodiment>
When the audio processing unit 6 of the first embodiment is used, the microphone unit 4 includes a directional microphone 4A and an omnidirectional microphone 4B that are arranged close to each other as shown in FIG. For example, the center distance between the directional microphone 4A and the omnidirectional microphone 4B is 2 cm. Furthermore, the directional microphone 4A and the omnidirectional microphone 4B are arranged so that the directional microphone 4A has the unidirectional pattern P1 shown in FIG. 4 and the omnidirectional microphone 4B has the omnidirectional pattern P2 shown in FIG. Place. Note that the directional characteristics shown in FIG. 4 (unidirectional pattern P1, omnidirectional pattern P2, and new directional pattern P3 obtained by voice processing) represent microphone sensitivities by sound arrival direction. The farther a certain point to be formed is from the center O, the higher the microphone sensitivity with respect to the sound from the direction from the certain point toward the center O is.

  As shown in FIG. 5, the audio processing unit 6 of the first embodiment includes FFT units 23 and 24, a power comparison spectrum determination unit 25, a memory unit 26, an unnecessary spectrum removal unit 27, and an IFFT unit 28. Prepare.

  The FFT unit 23 samples the output signal of the directional microphone 4A at 48 kHz and converts it into a digital signal, and then converts it into a frequency domain signal S1 [F] by FFT processing every 2048 samples. S 1 [F] is output to the power comparison spectrum determination unit 25 and the unnecessary spectrum removal unit 27. Further, the FFT unit 24 samples the output signal of the omnidirectional microphone 4B at 48 kHz and converts it into a digital signal, and then converts it into a frequency domain signal S2 [F] by FFT processing every 2048 samples. The region signal S 2 [F] is output to the power comparison spectrum determination unit 25.

  The power comparison spectrum determination unit 25 uses the power of the frequency domain signal S1 [F] and the power of the frequency domain signal S2 [F] in a predetermined frequency band (for example, the frequency of the FFT processing target in the FFT units 23 and 24). Area), the power of the frequency domain signal S1 [F] is compared with the power of the frequency domain signal S2 [F], and the relative power parameter (here, the frequency domain signal S1 [F] is compared). ] Is obtained for each frequency by subtracting the power of the signal S2 [F] in the frequency domain from the power of The memory unit 26 stores in advance a threshold value TH that is a value determined from the unidirectional pattern P1 and the omnidirectional pattern P2 shown in FIG. 4 and corresponding to the vector v1 shown in FIG. The power comparison spectrum determination unit 25 determines whether or not the relative power parameter is greater than the threshold value TH for each frequency, and determines that the sound component is from the front direction when the relative power parameter is greater than the threshold value TH.

  Based on the determination result of the power comparison spectrum determination unit 25, the unnecessary spectrum removal unit 27 removes unnecessary components that are not sound components from the front direction on the frequency domain from the signal S1 [F] in the frequency domain. The frequency domain signal from which the component has been removed is output to the IFFT unit 28. The IFFT unit 28 converts the output signal of the unnecessary spectrum removing unit 27 into a signal in the time domain by IFFT processing, and outputs it to the compression processing unit 7 (see FIG. 1) as the output signal of the audio processing unit 6 of the first embodiment. To do.

  A new directivity pattern P3 can be obtained by the sound processing as described above. In the voice processing unit 6 of the first embodiment, since the direction determination is performed using the relative power parameter, there is no restriction on the lower limit / upper limit frequency, and the direction determination for the wideband voice signal can be performed with high accuracy. Further, for example, by using the microphone unit 4 having directivity characteristics as shown in FIG. 4, the change in the relative power parameter due to the difference in the sound source direction can be increased. Direction determination can be performed with high accuracy.

  Note that in the imaging device, it is assumed that the positional relationship between the main sound source and the microphone unit changes depending on the shooting situation, but no matter how the positional relationship between the main sound source and the microphone unit changes, as described above, for example, By using the directivity characteristics as shown in FIG. 4, the change in relative power parameter due to the difference in sound source direction can be increased, so no problem occurs. In the present embodiment, sound from the front direction is targeted for determination, but by changing the threshold value TH stored in the memory unit 26 in advance, it is also possible to determine sound from other specific directions. It is. Furthermore, a specific direction can be switched by preparing a plurality of threshold TH values and selecting a value to be used for determination from among them.

Second Embodiment
When the audio processing unit 6 of the second embodiment is used, the microphone unit 4 includes a directional microphone 4A and omnidirectional microphones 4B and 4C that are arranged close to each other as shown in FIG. For example, the center interval between the directional microphone 4A and the omnidirectional microphone 4B and the center interval between the omnidirectional microphone 4B and the omnidirectional microphone 4C are each 2 cm. Furthermore, the directional microphone 4A and the omnidirectional are set such that the directional microphone 4A has the unidirectional pattern P1 shown in FIG. 4 and the omnidirectional microphones 4B and 4C have the omnidirectional pattern P2 shown in FIG. The sex microphones 4B and 4C are arranged.

  As shown in FIG. 7, the sound processing unit 6 of the first embodiment includes FFT units 29 and 31, HPFs (High Pass Filters) 32 and 33, LPFs (Low Pass Filters) 34 and 35, and a power comparison spectrum. A determination unit 36, a phase comparison spectrum determination unit 37, a memory unit 38, an unnecessary spectrum removal unit 39, and an IFFT unit 40 are provided. The cutoff frequencies of the HPFs 32 and 33 and the cutoff frequencies of the LPFs 34 and 35 are both 8 kHz.

  The FFT unit 29 samples the output signal of the directional microphone 4A at 48 kHz and converts it into a digital signal, and then converts it into a frequency domain signal S1 [F] by FFT processing every 2048 samples. S1 [F] is output to the unnecessary spectrum removal unit 39 and the power comparison spectrum determination unit 36 via the HPF 32. Further, the FFT unit 30 samples the output signal of the omnidirectional microphone 4B at 48 kHz and converts it into a digital signal, and then converts it into a frequency domain signal S2 [F] by FFT processing every 2048 samples. The region signal S2 [F] is output to the power comparison spectrum determination unit 36 via the HPF 33 and to the phase comparison spectrum determination unit 37 via the LPF 34. Further, the FFT unit 31 samples the output signal of the omnidirectional microphone 4C at 48 kHz and converts it into a digital signal, and then converts it into a frequency domain signal S3 [F] by FFT processing every 2048 samples. The region signal S3 [F] is output to the phase comparison spectrum determination unit 37 via the LPF 35.

  The power comparison spectrum determination unit 36 calculates, for each frequency, the power of the frequency component of 8 kHz or higher in the frequency domain signal S1 [F] and the power of the frequency component of 8 kHz or higher in the frequency domain signal S2 [F]. The power of the frequency component of 8 kHz or more in the signal S1 [F] in the region is compared with the power of the frequency component of 8 kHz or more in the signal S2 [F] in the frequency region, and the relative power parameter (here, the signal S1 in the frequency region is compared). The frequency component power of 8 kHz or more in [F] is subtracted from the power of the frequency component of 8 kHz or more in the signal S2 [F] in the frequency domain) for each frequency.

  The phase comparison spectrum determination unit 37 calculates, for each frequency, the phase of the frequency component of 8 kHz or less in the frequency domain signal S2 [F] and the phase of the frequency component of 8 kHz or less in the frequency domain signal S3 [F] for each frequency. The phase of the frequency component of 8 kHz or less in the domain signal S2 [F] is compared with the phase of the frequency component of 8 kHz or less in the frequency domain signal S3 [F], and the relative phase parameter (here, the signal S2 in the frequency domain) is compared. A value obtained by subtracting the phase of the frequency component of 8 kHz or less in the frequency domain signal S3 [F] from the phase of the frequency component of 8 kHz or less in [F] is obtained for each frequency.

The memory unit 26 has a threshold TH that is a value determined from the unidirectional pattern P1 and the omnidirectional pattern P2 shown in FIG. 4 and that corresponds to the vector v1 shown in FIG. 4, and an angle shown in FIG. A threshold value θ that is a value corresponding to α ° is stored in advance. The threshold θ is expressed by the following formula. Here, F represents frequency and V represents sound velocity.
θ = 2.0π × F × 0.02 cos (90−α) ° / V

  The power comparison spectrum determination unit 36 determines for each frequency whether or not the relative power parameter is larger than the threshold value TH for a frequency component of 8 kHz or more. If the relative power parameter is larger than the threshold value TH, the sound from the front direction is determined. Determined to be a component. Further, the phase comparison spectrum determination unit 37 determines, for each frequency, whether or not the absolute value of the relative phase parameter is smaller than the threshold value θ for a frequency component of 8 kHz or less, and the absolute value of the relative phase parameter is smaller than the threshold value θ. When it is small, it is determined that the sound component is from the front direction.

  Based on the determination result of the power comparison spectrum determination unit 36 and the determination result of the phase comparison spectrum determination unit 37, the unnecessary spectrum removal unit 39 removes an unnecessary component that is not a sound component from the front direction from the signal S1 [F] in the frequency domain. A signal in the frequency domain that has been removed on the frequency domain and from which unnecessary components have been removed is output to IFFT section 40. The IFFT unit 30 converts the output signal of the unnecessary spectrum removing unit 39 into a signal in the time domain by IFFT processing, and outputs the signal to the compression processing unit 7 (see FIG. 1) as the output signal of the audio processing unit 6 of the second embodiment. To do.

  The audio processing unit 6 according to the second embodiment performs direction determination using the relative phase parameter in the low frequency band where the direction determination can be performed accurately using the relative phase parameter. In the high frequency band where the direction determination cannot be performed with high accuracy, the same audio processing as that of the audio processing unit 6 of the first embodiment is performed, and thus the same effect as that of the audio processing unit 6 of the first embodiment is obtained. .

<Third Embodiment>
The sound processing unit 6 of the first embodiment and the second embodiment determines a sound in the front direction, but the sound processing unit 6 of the third embodiment determines a sound in two directions (Lch, Rch). When using the audio processing unit 6 of the third embodiment, the configuration of the microphone unit 4 and the arrangement of each microphone are the same as when using the audio processing unit 6 of the third embodiment (see FIGS. 3 and 4).

  As shown in FIG. 8, the sound processing unit 6 of the third embodiment includes FFT units 41 and 42, a power comparison spectrum determination unit 43, a phase comparison spectrum determination unit 44, a memory unit 45, and an unnecessary spectrum removal unit. 46 and 47, and IFFT units 48 and 49.

  Next, the operation of the voice processing unit 6 of the third embodiment will be described with reference to the flowchart shown in FIG.

  In step # 101, the FFT unit 41 samples the output signal of the directional microphone 4A at 48 kHz and converts it into a digital signal, and then converts it into a frequency domain signal S1 [F] by FFT processing every 2048 samples. The frequency domain signal S 1 [F] is output to the power comparison spectrum determination unit 43 and the phase comparison spectrum determination unit 44. In step # 10, the FFT unit 42 samples the output signal of the omnidirectional microphone 4B at 48 kHz and converts it into a digital signal, and then converts it into a frequency domain signal S2 [F] by FFT processing every 2048 samples. Then, the signal S2 [F] in the frequency domain is output to the power comparison spectrum determination unit 43, the phase comparison spectrum determination unit 44, and the unnecessary spectrum removal units 46 and 47.

  In subsequent step # 102, the power comparison spectrum determination unit 43 and the phase comparison spectrum determination unit 44 set the processing target frequency f to the minimum value.

  In subsequent step # 103, the power comparison spectrum determination unit 43 determines the power PW1 [f] at the processing target frequency f component of the frequency domain signal S1 [F] and the processing target frequency f of the frequency domain signal S2 [F]. The power PW2 [f] at the component is calculated. Also, in step # 30, the phase comparison spectrum determination unit 44 performs processing on the phase PH1 [f] at the processing target frequency f component of the frequency domain signal S1 [F] and the processing target frequency of the frequency domain signal S2 [F]. The phase PH2 [f] at the f component is calculated.

In subsequent step # 104, the power comparison spectrum determination unit 43 compares the power PW1 [f] with the power PW2 [f] to obtain a relative power parameter (PW1 [f] −PW2 [f]), and the relative It is determined whether or not the power parameter is within a range of ± 0.5 dB of threshold values TH _{L, R} (values corresponding to the vectors v _L and v _R shown in FIG. 4) stored in the memory unit 26 in advance. .

Relative power parameter (PW1 [f] -PW2 [f ]) is (threshold TH _L, R -0.5dB) or more (threshold TH _L, R + 0.5dB) not more than (YES in Step # 104), L The sound component from the direction and the sound component from the R direction are determined (step # 105), and the process proceeds to step # 107. On the other hand, the relative power parameter (PW1 [f] −PW2 [f]) is not (threshold TH _{L, R} −0.5 dB) or more (threshold TH _{L, R} +0.5 dB) or less (NO in step # 104). The sound component from the L direction and the sound component from the R direction are determined to be unnecessary sounds (step # 106), and the process proceeds to step # 112.

In step # 107, the phase comparison spectrum determination unit 44 compares the phase PH1 [f] with the phase PH2 [f] to obtain a relative phase parameter (PH1 [f] −PH2 [f]), and calculates the relative phase. The absolute value of the difference between the parameter and the threshold value θ _L (a value corresponding to the angle β _L ° shown in FIG. 4) stored in advance in the memory unit 26, its relative phase parameter, and the memory unit 26 are stored in advance. The magnitude relation with the absolute value of the difference from the threshold value θ _R (value corresponding to the angle β _R ° shown in FIG. 4) is determined. The threshold values θ _L and θ _R are each expressed by the following equations. Here, F represents frequency and V represents sound velocity.
θ _L = 2.0π × F × 0.02 cos (90−β _L ) ° / V
θ _R = 2.0π × F × 0.02 cos (90 + β _R ) ° / V

If the absolute value of the difference between the relative phase parameter and the threshold value θ _L is equal to or smaller than the absolute value of the difference between the relative phase parameter and the threshold value θ _R (YES in step # 107), it is determined that the sound component is from the L direction. (Step # 108), the process proceeds to Step # 110. On the other hand, if the absolute value of the difference between the relative phase parameter and the threshold value θ _L is larger than the absolute value of the difference between the relative phase parameter and the threshold value θ _R (NO in step # 107), it is determined that the sound component is from the R direction. (Step # 109), the process proceeds to Step # 111.

  In Step # 110, the unnecessary spectrum removing unit 46 sets SL [f] = S2 [f], and the unnecessary spectrum removing unit 47 sets SR [f] = 0, and the process proceeds to Step # 113.

  In Step # 111, the unnecessary spectrum removing unit 46 sets SL [f] = 0, and the unnecessary spectrum removing unit 47 sets SR [f] = S2 [f], and the process proceeds to Step # 113.

  In Step # 112, the unnecessary spectrum removing unit 46 sets SL [f] = 0, and the unnecessary spectrum removing unit 47 sets SR [f] = 0, and the process proceeds to Step # 113.

  In step # 113, the power comparison spectrum determination unit 43 and the phase comparison spectrum determination unit 44 determine whether the processing target frequency f is set to the maximum value (for example, 24 kHz). If the processing target frequency f is not set to the maximum value (for example, 24 kHz) (NO in step # 113), the processing target frequency f is updated and set to a value larger by one step (step # 114). Return. On the other hand, if the processing target frequency f is set to the maximum value (for example, 24 kHz) (YES in step # 113), SL [F] that is an aggregate of SL [f] of each processing target frequency is generated by the IFFT unit 48. The IFFT process is performed to convert the signal into a time domain signal, which is output to the compression processing unit 7 (see FIG. 1) as an Lch output signal of the audio processing unit 6 of the third embodiment, and a set of SR [f] of each processing target frequency SR [F], which is a body, is IFFT processed by the IFFT unit 49 and converted into a signal in the time domain, and is output to the compression processing unit 7 (see FIG. 1) as an Rch output signal of the audio processing unit 6 of the third embodiment. (Step # 115).

  The sound processing unit 6 of the third embodiment has the feature that it can determine sound in two directions in addition to the same effects as the sound processing unit 6 of the first embodiment. Further, by increasing the number of thresholds related to the relative power parameter and relative phase parameter stored in advance in the memory unit 45 as compared with the present embodiment, it is possible to determine sounds in three or more directions.

<Fourth embodiment>
When the audio processing unit 6 according to the fourth embodiment is used, the microphone unit 4 includes a directional microphone 4D and a directional microphone 4E that are arranged close to each other as shown in FIG. For example, the center distance between the directional microphone 4D and the directional microphone 4E is 2 cm. Furthermore, more specifically, the directional microphone 4D has the unidirectional pattern P4 shown in FIG. 11 so that the directional microphone 4D and the directional microphone 4E have different directivity characteristics, and the directional microphone 4E. The directional microphone 4D and the omnidirectional microphone 4E are arranged so as to have the unidirectional pattern P5 shown in FIG. Note that the directivity characteristics (unidirectional patterns P4 and P5) shown in FIG. 11 represent the microphone sensitivity according to the sound arrival direction, and the more a certain point forming the pattern is from the center O, the more This shows that the microphone sensitivity with respect to the sound from the direction from the point toward the center O is high.

  As shown in FIG. 12, the audio processing unit 6 according to the fourth embodiment includes FFT units 50 and 51, a power comparison spectrum determination unit 52, an unnecessary spectrum removal unit 53, an IFFT unit 54, and a gain α. A gain adjuster 55, gain adjusters 56 and 57 having a gain of (1-α), and adders 58 and 59 are provided.

  The FFT unit 50 samples the output signal of the directional microphone 4D at 48 kHz and converts it into a digital signal, and then converts it into a frequency domain signal S1 [F] by FFT processing every 2048 samples. S 1 [F] is output to the power comparison spectrum determination unit 52. Further, the FFT unit 51 samples the output signal of the directional microphone 4E at 48 kHz and converts it into a digital signal, and then converts it into a frequency domain signal S2 [F] by FFT processing every 2048 samples. The signal S2 [F] is output to the power comparison spectrum determination unit 52 and the unnecessary spectrum removal unit 53.

  The power comparison spectrum determination unit 52 calculates the power of the frequency domain signal S1 [F] and the power of the frequency domain signal S2 [F] for each frequency, and the power of the frequency domain signal S1 [F] and the frequency domain. It is determined for each frequency whether or not the power of the signal S2 [F] matches, and if it matches, the sound component from the front direction is determined.

  Based on the determination result of the power comparison spectrum determination unit 52, the unnecessary spectrum removal unit 53 removes an unnecessary component that is not a sound component from the front direction on the frequency domain from the signal S2 [F] in the frequency domain. The frequency domain signal from which the component is removed is output to IFFT section 54. The IFFT unit 54 converts the output signal of the unnecessary spectrum removal unit 53 into a signal in the time domain by IFFT processing.

  The output signal of the IFFT unit 54 is gain-adjusted by the gain adjuster 55, mixed with the output signal of the directional microphone 4D gain-adjusted by the gain adjuster 56 in the adder 58, and then the sound processing unit of the fourth embodiment. 6 Lch output signals are output to the compression processing unit 7 (see FIG. 1).

  The output signal of the IFFT unit 54 is gain-adjusted by the gain adjuster 55, mixed with the output signal of the directional microphone 4E gain-adjusted by the gain adjuster 57 by the adder 59, and then the sound of the fourth embodiment. The Rch output signal of the processing unit 6 is output to the compression processing unit 7 (see FIG. 1).

  For example, the gain adjusters 55 to 57 change the value of α in conjunction with the camera zoom information output from the CPU 17 (see FIG. 1). α is set to 0.0.

  The voice processing unit 6 of the fourth embodiment has the feature that the sound enhancement in the front direction can be changed in addition to the same effects as the voice processing unit 6 of the first embodiment. .

  The above-described imaging apparatus shown in FIG. 1 is an imaging apparatus equipped with a sound collection environment determination apparatus applied when recording a sound signal collected by the determination method according to the present invention. However, the determination according to the present invention is performed. Since the method uses the collected audio signal to determine the direction of the audio signal, there is no need to make a determination when recording the collected audio signal, and playback is performed using the collected audio signal. It is also possible to determine the direction with respect to the audio signal, that is, to record / reproduce the collected audio signal and to make the determination using the reproduced audio signal. In other words, in the present invention, the timing for determining the direction of the audio signal is not limited to recording the collected audio signal, but may be when reproducing the audio signal. As described above, in the determination method according to the present invention, the timing for performing the direction determination on the audio signal is not limited to recording the collected audio signal, so that it depends on other processing performed using video and audio information. Thus, it is possible to determine the direction of the audio signal when recording the collected audio signal or when reproducing the recorded audio signal.

  Hereinafter, an imaging apparatus equipped with a determination apparatus to which the determination method according to the present invention is applied during reproduction will be described.

  FIG. 13 is a block diagram showing another example of the internal configuration of an imaging apparatus equipped with the determination apparatus according to the present invention. In FIG. 13, parts that are substantially the same as those in FIG.

  The imaging apparatus shown in FIG. 13 is different from the imaging apparatus shown in FIG. 1 in that an audio processing unit 6a is provided instead of the audio processing unit 6, and audio processing is performed between the expansion processing unit 9 and the audio output circuit unit 13. The point is that a portion 6b is provided.

  Unlike the audio processing unit 6, the audio processing unit 6a performs A / D conversion on an audio signal that is an analog signal from the microphone unit 4, but the determination method according to the present invention and the audio corresponding to the determination result No processing is performed.

  The audio processing unit 6b has the same configuration as the audio processing unit 6 except that the FFT unit does not perform A / D conversion. Since the audio processing performed in the audio processing unit 6b is basically the same as the audio processing performed in the audio processing unit 6, the description thereof is omitted here.

  In addition, since the present invention relates to direction determination for an audio signal, a block related to a video is not essential. Therefore, the present invention can also be applied to electronic devices other than the imaging device, for example, an audio recording device, an audio reproducing device, an audio recording / reproducing device (for example, an IC recorder), and the like.

  In the first to fourth embodiments described above, the threshold value is stored in the memory unit in the voice processing unit 6. However, the threshold value may be stored in the memory 18 and the memory unit in the voice processing unit 6 may be abolished. .

  The first to fourth embodiments described above can be implemented in appropriate combination. For example, the determination method of the first embodiment and the determination method of the third embodiment are combined, and the sound in the front direction obtained by the determination method of the first embodiment is adjusted with the gain α, and the third embodiment The L direction sound obtained by the determination method is mixed with the gain adjusted by the gain (1-α) to generate an Lch output signal, and the front direction sound obtained by the determination method of the first embodiment is generated. It is possible to generate an Rch output signal by mixing the gain adjusted with the gain α and the R direction sound obtained by the determination method of the third embodiment and adjusting the gain with the gain (1-α). It is.

  In the first to fourth embodiments described above, the sound component from the specific direction is determined as the direction determination for the audio signal. However, the present invention is not limited to this, and the relative power obtained for each frequency is determined. The sound source direction may be determined based on the parameter. As a method of using the determination result of the sound source direction, for example, a method of using in a video conference system and controlling the camera according to the determination result of the sound source direction so that the camera faces the sound source direction (speaker direction). Can be considered.

  The present invention makes an electronic device that stores and / or reproduces an audio signal (for example, an imaging apparatus, an IC recorder, a portable device equipped with those functions, or a computer function as a means for storing and / or reproducing an audio signal) For example, a computer that is operated by software).

These are block diagrams which show the example of 1 internal structure of the imaging device carrying the sound collection environment determination apparatus which concerns on this invention. These are the flowcharts for demonstrating the basic operation | movement at the time of video recording of the imaging device shown in FIG. These are figures which show the directivity characteristic of the microphone part in 1st Embodiment. These are figures which show arrangement | positioning of each microphone of the microphone part in 1st Embodiment. These are block diagrams which show the structure of the audio | voice processing part of 1st Embodiment. These are figures which show arrangement | positioning of each microphone of the microphone part in 2nd Embodiment. These are block diagrams which show the structure of the audio | voice processing part of 2nd Embodiment. These are block diagrams which show the structure of the audio | voice processing part of 3rd Embodiment. These are the operation | movement flowcharts of the audio | voice processing part of 3rd Embodiment. These are figures which show the directivity characteristic of the microphone part in 4th Embodiment. These are figures which show arrangement | positioning of each microphone of the microphone part in 4th Embodiment. These are block diagrams which show the structure of the audio | voice processing part of 4th Embodiment. These are block diagrams which show the other internal structural example of the imaging device carrying the determination apparatus which concerns on this invention. These are block diagrams which show the structure of the audio | voice processing part for implement | achieving the conventional specific sound source emphasis method. These are figures which show the positional relationship of a microphone and a sound source.

Explanation of symbols

1 Solid-state image sensor (image sensor)
2 Lens part 3 AFE
DESCRIPTION OF SYMBOLS 4 Microphone part 4A Directional microphone 4B Omnidirectional microphone 5 Image processing part 6, 6a, 6b Audio processing part 7 Compression processing part 8 Driver part 9 Decompression processing part 10 Video output circuit part 11 Video output terminal 12 Display part 13 Audio output Circuit unit 14 Audio output terminal 15 Speaker unit 16 Timing generator (TG)
17 CPU
18 Memory 19 Operation unit 20, 21 Bus line 22 External memory 23, 24, 29-31, 41, 42, 50, 51 FFT unit 25, 36, 43, 52 Power comparison spectrum determination unit 26, 38, 45 Memory unit 27 , 39, 46, 47, 53 Unnecessary spectrum removing unit 28, 40, 48, 49, 54 IFFT unit 32, 33 HPF
34, 35 LPF
37, 44 Phase comparison spectrum determination unit 55-57 Gain adjuster 58, 59 Adder P1, P4, P5 Unidirectional pattern P2 Non-directional pattern P3 New directivity pattern obtained by voice processing

Claims (9)

A first time-frequency converter that converts the output signal of the first microphone to time-frequency;
A second time-frequency conversion unit that performs time-frequency conversion of an output signal of a second microphone having a directivity characteristic different from that of the first microphone;
The power of the frequency domain signal output from the first time frequency converter is compared with the power of the frequency domain signal output from the second time frequency converter for each frequency in a predetermined frequency band. A power comparator,
And a determination unit that determines a sound or a sound source direction from a specific direction using a comparison result in the power comparison unit. A storage unit for storing a determination condition based on a difference in directivity between the first microphone and the second microphone;
The determination device according to claim 1, wherein the determination unit determines a sound from a specific direction based on a comparison result in the power comparison unit and a determination condition stored in the storage unit. The predetermined frequency band is a first frequency band;
A third time-frequency conversion unit for time-frequency converting the output signal of the third microphone having the same directivity characteristics as the second microphone;
The phase of the frequency domain signal output from the second time frequency converter and the phase of the frequency domain signal output from the third time frequency converter in a band lower than the first frequency band. A phase comparator for comparing each frequency in a certain second frequency band;
A first storage unit that stores a first determination condition based on a difference in directivity between the first microphone and the second microphone;
A second storage unit that stores a second determination condition based on a positional relationship between the second microphone and the third microphone;
The determination unit determines sound from a specific direction of the first frequency band from a comparison result in the power comparison unit and a first determination condition stored in the first storage unit, and the phase The sound from a specific direction of the second frequency band is determined from a comparison result in the comparison unit and a second determination condition stored in the second storage unit. Judgment device. The phase of the frequency domain signal output from the first time frequency converter is compared with the phase of the frequency domain signal output from the second time frequency converter for each frequency in the predetermined frequency band. A phase comparator to
A first storage unit that stores a first determination condition based on a difference in directivity between the first microphone and the second microphone;
A second storage unit that stores a second determination condition based on a positional relationship between the first microphone and the second microphone;
The determination unit is either a sound from the first direction or a sound from the second direction based on the comparison result in the power comparison unit and the first determination condition stored in the first storage unit. If the primary determination unit that determines whether the sound is a sound from the first direction or the sound from the second direction is determined by the primary determination unit and the primary determination unit, the phase comparison A secondary determination unit that determines whether the sound is from the first direction based on the comparison result in the unit and the second determination condition stored in the second storage unit. The determination apparatus according to claim 1, characterized in that: The directivity pattern of the first microphone and the directivity pattern of the second microphone are symmetrical,
Comparison result that the power of the frequency domain signal output from the first time frequency converter is equal to the power of the frequency domain signal output from the second time frequency converter by the power comparator. The determination device according to claim 1, wherein the determination unit determines that the sound is from the front direction when the sound is obtained. Comprising at least one determination device according to claim 1,
An electronic apparatus that performs sound processing on a collected sound signal based on a determination result of the determination device. It has a recording / playback function for collected audio signals,
The electronic device according to claim 6, wherein the determination device performs determination processing either when recording the collected audio signal or when reproducing the recorded audio signal.   The electronic apparatus according to claim 6, wherein the electronic apparatus is an imaging apparatus including a camera that captures an image. A first time-frequency conversion step of time-frequency converting the output signal of the first microphone;
A second time-frequency conversion step for time-frequency converting an output signal of a second microphone having a directivity characteristic different from that of the first microphone;
Power comparison for comparing the frequency domain signal power obtained by the first time frequency conversion step and the frequency domain signal power obtained by the second time frequency conversion step for each frequency in a predetermined frequency band. Steps,
And a determination step of determining a sound or sound source direction from a specific direction using a comparison result obtained by the power comparison step.

Images