JP2012168477A - Noise estimation device, signal processor, imaging apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately reduce noise overlapping a sound signal.SOLUTION: A noise estimation device (150) includes: a calculation part (232) for calculating the degree of noise similarity indicating the degree of similarity between a sound signal and noise on the basis of the frequency spectrum of an input sound signal and the frequency spectrum of noise; and a noise estimation part (233) for estimating an estimated noise included in the sound signal on the basis of the degree of noise similarity calculated by the calculation part (232).

Description

  The present invention relates to a noise estimation device, a signal processing device, an imaging device, and a program.

  A technique for reducing noise superimposed on a sound signal is known (see, for example, Patent Document 1). In the technique described in Non-Patent Document 1, stationary noise superimposed on a sound signal is reduced by a predetermined estimated noise.

BOLL, S. F. "Suppression of Acoustic Noise in Speech Using Spectral Subtraction." IEEE TRANSACTION ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, vol. ASSP-27, pp. 113-120, APRIL, 1979.

However, in the technique described in Non-Patent Document 1, for example, when noise whose magnitude is unsteady is reduced, there is a difference between noise actually mixed in the sound signal and estimated noise. In some cases, excessive deterioration or excessive subtraction of noise may cause deterioration of sound or residual noise. Further, for example, when noise generated intermittently is reduced, noise may be excessively subtracted in a portion where noise is not mixed, resulting in sound deterioration.
That is, the technique described in Non-Patent Document 1 has a problem that noise superimposed on the sound signal cannot be reduced appropriately.

  The present invention has been made to solve the above problems, and an object thereof is to provide a noise estimation device, a signal processing device, an imaging device, and a program capable of appropriately reducing noise superimposed on a sound signal. There is to do.

  In order to solve the above problem, the present invention calculates a noise similarity indicating the degree of similarity between the sound signal and the noise based on the frequency spectrum of the input sound signal and the frequency spectrum of the noise. A noise estimation device comprising: a calculation unit; and a noise estimation unit that estimates an estimated noise included in the sound signal based on the noise similarity calculated by the calculation unit.

  In addition, the present invention includes the above noise estimation device, and a noise reduction processing unit that reduces noise included in the sound signal based on the estimated noise estimated by the noise estimation device. It is a signal processing device.

  Moreover, this invention is an imaging device provided with said signal processing apparatus.

  Further, according to the present invention, in a computer as a noise estimation device, the calculation unit indicates the degree of similarity between the sound signal and the noise based on the frequency spectrum of the input sound signal and the frequency spectrum of the noise. A calculation procedure for calculating a noise similarity, and a noise estimation unit for executing a noise estimation procedure for estimating an estimated noise included in the sound signal based on the noise similarity calculated by the calculation procedure It is a program.

  According to the present invention, it is possible to appropriately reduce noise superimposed on a sound signal.

It is a schematic block diagram which shows the imaging device by this embodiment. It is a schematic block diagram which shows the signal processing apparatus in the embodiment. It is a conceptual diagram which shows an example of the input frequency feature vector and noise frequency feature vector in the embodiment. It is a flowchart which shows the operation | movement of the noise reduction process in the embodiment. It is explanatory drawing which shows the example of the sound signal with which the noise in the same embodiment was superimposed. It is explanatory drawing explaining an example of the generation method of the frequency spectrum of the noise in the embodiment.

Hereinafter, a signal processing apparatus and an imaging apparatus according to an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a schematic block diagram illustrating the imaging apparatus 1 according to the present embodiment.
In this figure, the imaging apparatus 1 according to the present embodiment includes an imaging unit 10, a buffer memory unit 30, an image processing unit 40, a display unit 50, a storage unit 60, a communication unit 70, an operation unit 80, and a CPU (Central Processing Unit) 90. , A microphone 21, an A / D (Analog / Digital) converter 22, a sound signal processor 23, and a bus 300. Among the configurations of the imaging apparatus 1, for example, the sound signal processing unit 23 and a part of the storage unit 60 correspond to the signal processing apparatus 100. For example, a part of the sound signal processing unit 23 and a part of the storage unit 60 correspond to the noise estimation device 150.

  The imaging unit 10 includes an optical system 11, an imaging element 19, and an A / D conversion unit 20. The imaging unit 10 is controlled by the CPU 90 in accordance with the set imaging conditions (for example, an aperture value, an exposure value, etc.). An image is formed on the image sensor 19 and image data based on the optical image converted into a digital signal by the A / D converter 20 is generated.

The optical system 11 includes a focus adjustment lens (hereinafter referred to as an AF (Auto Focus) lens) 12, a camera shake prevention lens (hereinafter referred to as a VR (Vibration Reduction) lens) 13, a zoom lens 14, a zoom encoder 15, A lens driving unit 16, an AF encoder 17, and a camera shake preventing unit 18 are provided.
The optical system 11 guides the optical image that has passed through the zoom lens 14, the VR lens 13, and the AF lens 12 to the light receiving surface of the image sensor 19.

The lens driving unit 16 controls the position of the zoom lens 14 or the AF encoder 17 based on a drive control signal input from a CPU 90 described later.
The camera shake prevention unit 18 controls the position of the VR lens 13 based on a drive control signal input from a CPU 90 described later. The camera shake prevention unit 18 may detect the position of the VR lens 13.

The zoom encoder 15 detects a zoom position representing the position of the zoom lens 14 and outputs the detected zoom position to the CPU 90.
The AF encoder 17 detects a focus position representing the position of the AF lens 12 and outputs the detected zoom position and focus position to the CPU 90.

  The optical system 11 described above may be attached to and integrated with the imaging apparatus 1 or may be attached to the imaging apparatus 1 so as to be detachable.

For example, the imaging element 19 converts an optical image formed on the light receiving surface into an electrical signal and outputs the electrical signal to the A / D conversion unit 20.
In addition, the image sensor 19 uses the image data obtained when a shooting instruction is received via the operation unit 80 as shot image data of a shot still image via the A / D conversion unit 20 and the image processing unit 40. And stored in the storage medium 200.

  On the other hand, for example, the imaging device 19 uses continuously obtained image data as through image data through the A / D conversion unit 20 and the image processing unit 40 in a state where an imaging instruction is not received via the operation unit 80. To the CPU 90 and the display unit 50.

  The A / D converter 20 performs analog / digital conversion on the electronic signal converted by the image sensor 19 and outputs image data that is the converted digital signal.

The buffer memory unit 30 temporarily stores the image data captured by the imaging unit 10, the sound signal converted by the sound signal processing unit 23, and the like.
The image processing unit 40 refers to the image processing conditions stored in the storage unit 60 and performs image processing on the image data recorded in the buffer memory unit 30 or the storage medium 200.

  The display unit 50 is, for example, a liquid crystal display, and displays image data obtained by the imaging unit 10, an operation screen, and the like.

The storage unit 60 stores determination conditions referred to when scene determination is performed by the CPU 90, imaging conditions, and the like. The storage unit 60 also stores information used for noise reduction processing for reducing noise in the sound signal in the sound signal processing unit 23 described later. Here, the information used for the noise reduction processing is, for example, a noise frequency feature vector described later.
The storage unit 60 includes a noise frequency feature vector storage unit 61.

  The noise frequency feature vector storage unit 61 stores a noise frequency feature vector to be described later. The noise frequency feature vector is stored in advance, for example, when the imaging apparatus 1 is manufactured or shipped. Further, the noise frequency feature vector storage unit 61 may store a plurality of noise frequency feature vectors according to the type of the corresponding noise. For example, noise frequency feature vectors corresponding to noise caused by any one of the mechanical sounds of the zoom lens 14, the VR lens 13, and the AF lens 12 are stored in the noise frequency feature vector storage unit 61. May be.

The microphone 21 collects sound and outputs a sound signal corresponding to the collected sound. This sound signal is an analog signal.
The A / D conversion unit 22 performs analog-to-digital conversion of a sound signal that is an analog signal input from the microphone 21 into a sound signal that is a digital signal.

The sound signal processing unit 23 performs sound signal processing such as noise reduction on the sound signal converted into a digital signal by the A / D conversion unit 22 and stores the sound signal that has been processed. Store in the medium 200. The sound signal processing unit 23 includes a noise reduction processing unit 24, a frequency feature vector generation unit 231, an inner product calculation unit 232, and a noise estimation unit 233. Details of the sound signal processing unit 23 will be described later.
When the sound signal processed by the sound signal processing unit 23 is stored in the storage medium 200, the sound signal may be stored in a temporal relationship with the image data captured by the image sensor 19. You may memorize | store as a moving image containing a sound signal.

The communication unit 70 is connected to a removable storage medium 200 such as a card memory, and performs writing, reading, or erasing of information on the storage medium 200.
The operation unit 80 includes, for example, a power switch, a shutter button, and other operation keys. When the operation unit 80 is operated by the user, the operation unit 80 receives a user operation input and outputs the operation input to the CPU 90.

  The storage medium 200 is a storage unit that is detachably connected to the imaging device 1. For example, image data generated (captured) by the imaging unit 10 or sound signal processing by the sound signal processing unit 23. The recorded sound signal is stored.

  The CPU 90 controls the entire imaging apparatus 1. For example, the CPU 90 includes a zoom position input from the zoom encoder 15, a focus position input from the AF encoder 17, and an operation input input from the operation unit 80. Based on this, a drive control signal for controlling the positions of the zoom encoder 15 and the AF encoder 17 is generated. The CPU 90 controls the positions of the zoom encoder 15 and the AF encoder 17 via the lens driving unit 16 based on this drive control signal.

  The bus 300 includes an imaging unit 10, a sound signal processing unit 23, a buffer memory unit 30, an image processing unit 40, a display unit 50, a storage unit 60, a communication unit 70, an operation unit 80, and a CPU 90. The data output from each unit is transferred.

Next, detailed configurations of the signal processing device 100 and the sound signal processing unit 23 in the configuration of FIG. 1 will be described with reference to FIG.
FIG. 2 is a schematic block diagram showing the signal processing apparatus 100 in the present embodiment.
In this figure, the signal processing apparatus 100 includes a sound signal processing unit 23 and a noise frequency feature vector storage unit 61. Note that the noise frequency feature vector storage unit 61 is provided in the storage unit 60 and is a part of the storage unit 60, for example. Further, the signal processing device 100 includes a noise estimation device 150. Among the configurations of the signal processing device 100, for example, a frequency feature vector generation unit 231, an inner product calculation unit 232, a noise estimation unit 233, and a noise frequency feature. The vector storage unit 61 corresponds to the noise estimation device 150.

  The sound signal processing unit 23 performs sound signal processing such as noise reduction on the sound signal converted into a digital signal by the A / D conversion unit 22 and stores the sound signal that has been processed. Store in the medium 200. In the present embodiment, “noise” refers to a noise signal that is generated by the operation of the operation unit and is included in the sound signal (that is, superimposed on the sound signal). That is, “noise” is non-stationary noise.

The operation unit referred to here is also called a mechanism unit, and as an example, is the above-described zoom lens 14, VR lens 13, AF lens 12, or operation unit 80. The operation unit is a configuration in which sound is generated (or that sound may be generated) when operated or operated among the configurations included in the imaging apparatus 1.
In addition, the operation unit refers to a sound that is generated by operation or a sound that is generated by operation in the configuration of the imaging apparatus 1 (or is collected). It may be sounded).

The sound signal processing unit 23 performs sound signal processing such as noise reduction based on the control signal supplied from the CPU 90.
The sound signal processing unit 23 includes a frequency feature vector generation unit 231, an inner product calculation unit 232, a noise estimation unit 233, and a noise reduction processing unit 24.

  The frequency feature vector generation unit 231 converts the sound signal converted into the digital signal by the A / D conversion unit 22 into a frequency spectrum by performing Fourier transform (for example, FFT (Fast Fourier Transform) transform) on a frame basis. A frame is a section obtained by dividing a sound signal. Here, for example, a case will be described in which a frame is a section in which a predetermined period is shifted by a half period.

  Then, the frequency feature vector generation unit 231 generates an input frequency feature vector based on the converted frequency spectrum of the sound signal. That is, the frequency feature vector generation unit 231 converts the sound signal into a frequency spectrum, and generates an input frequency feature vector based on the converted frequency spectrum of the sound signal.

  Note that the input frequency feature vector is a vector generated with elements (for example, absolute values) corresponding to frequency components (frequency bins) in the frequency spectrum of the sound signal. Details of the input frequency feature vector will be described later with reference to FIG.

  Further, the frequency feature vector generation unit 231 supplies the generated input frequency feature vector to the inner product calculation unit 232 and the noise reduction processing unit 24.

  The inner product calculation unit 232 (calculation unit) calculates noise based on the inner product value of the input frequency feature vector supplied from the frequency feature vector generation unit 231 and the noise frequency feature vector read from the noise frequency feature vector storage unit 61. Calculate similarity. As an example, the inner product calculation unit 232 (calculation unit) calculates the inner product value of the input frequency feature vector and the noise frequency feature vector as the noise similarity. The noise similarity is information indicating the degree of similarity between a sound signal and noise. That is, the noise similarity is a value indicating how much noise is included (mixed) in the sound signal. The inner product calculation unit 232 supplies the calculated noise similarity to the noise estimation unit 233.

  Here, the noise frequency feature vector is a vector indicating the feature of noise generated based on the frequency spectrum of noise, and is stored in advance in the noise frequency feature vector storage unit 61. Further, the noise frequency feature vector is generated with an intensity (for example, absolute value) corresponding to each frequency component (frequency bin) in the frequency spectrum of noise as an element.

  That is, the noise frequency feature vector is generated based on the frequency spectrum of noise, and the input frequency feature vector is generated based on the frequency spectrum of the sound signal. Therefore, in other words, the inner product calculation unit 232 calculates the above-described noise similarity based on the frequency spectrum and the frequency spectrum of noise.

The noise frequency feature vector may be a normalized unit vector, for example. That is, assuming that the noise frequency feature vector before normalization is n ₀ , the normalized noise frequency feature vector n is expressed as Expression (1).

The noise estimation unit 233 estimates the estimated noise included in the sound signal based on the noise similarity calculated by the inner product calculation unit 232. That is, in this embodiment, the noise estimation unit 233 estimates the estimated noise based on the inner product value of the input frequency feature vector and the noise frequency feature vector n. Here estimated noise (vector ^~ n _k) is represented as formula (2). Here, the symbol “ ^˜ ” represents an estimated value, and “ ^˜ ” in the text represents a symbol attached immediately above the next character.

Here, k indicates a frame number, and vector X _k indicates an input frequency vector in the k-th frame.
The noise estimation unit 233 supplies the formula (2) estimating the noise calculated by (vector ^~ n _k) to the noise reduction processing unit 24.

Noise reduction processing unit 24, based on the input frequency feature vectors generated by the frequency feature vector generating unit 231, the estimated noise estimated by the noise estimating unit 233 (vector ^{~ n} _k) and the noise contained in the sound signal Execute processing to reduce. That is, the noise reduction processing unit 24 reduces noise included in the sound signal based on the estimated noise estimated by the noise estimation device 150. And the noise reduction process part 24 memorize | stores the sound signal which performed the process which reduces this noise in the storage medium 200. FIG.
In addition, the noise reduction processing unit 24 includes a noise subtraction unit 234 and an inverse conversion unit 235.

Noise subtraction unit 234, based on the input frequency characteristic vector X _k generated by the frequency feature vector generating unit 231, the estimated noise estimated by the noise estimating unit 233 (vector ^{~ n} _k) and is included in the sound signal noise is subtracted to calculate the estimated frequency characteristic vector ^~ S _k of the target sound obtained by subtracting the noise. Noise subtraction unit 234, the relational expression shown in equation (3) to calculate the estimated frequency characteristic vector ^~ S _k of the target sound. Here, the target sound is a target sound (sound to be recorded) that the user intends to pick up with the microphone 21 of the imaging device 1.

The noise subtracting unit 234 supplies the calculated estimated frequency feature vector ^~ S _k of the target sound to the inverse transform unit 235.

Inverse transform unit 235 returns the estimated frequency characteristic vector ^~ S _k of target sound calculated by the noise subtraction unit 234 into a frequency spectrum. Then, the inverse transform unit 235 performs inverse Fourier transform (for example, inverse FFT transform) on the time waveform based on the phase information of the sound signal mixed with noise, and synthesizes the sound signal of the target sound with reduced noise. Further, the inverse conversion unit 235 stores the generated sound signal in the storage medium 200 via the communication unit 70.
In addition, when a negative value is included in the frequency spectrum _obtained by converting the estimated frequency feature vector ^to Sk, the inverse transform unit 235 performs inverse Fourier transform by replacing the negative value with “0”.

Next, with reference to FIG. 3, the concept that the noise estimation apparatus 150 estimates the noise included in the sound signal will be described in detail.
FIG. 3 is a conceptual diagram showing an example of an input frequency feature vector and a noise frequency feature vector in the present embodiment. Here, in order to explain the concept in the present embodiment, a case where the number of frequency components (number of frequency bins) is two will be described.

  FIG. 3A shows an example of a frequency spectrum of noise and a frequency spectrum of a sound signal. In this graph, in each graph, the horizontal axis represents frequency components (frequency bins B1 and B2), and the vertical axis represents intensity (for example, PSD (Power Spectrum Density)).

  Moreover, in this figure, the frequency spectrum W1 has shown the frequency spectrum of noise. The frequency spectrum W2 indicates the frequency spectrum of the input sound signal (sound signal) x1, and the frequency spectrum W3 indicates the frequency spectrum of the input sound signal (sound signal) x2.

  FIG. 3B shows a vector space obtained by converting the frequency spectrum of noise and the frequency spectrum of the sound signal shown in FIG. In this figure, the horizontal axis indicates the intensity of the frequency bin B1, and the vertical axis indicates the intensity of the frequency bin B2.

  In this figure, a vector n represents a noise frequency feature vector (normalized noise frequency feature vector) corresponding to the noise frequency spectrum W1. The normalized noise frequency feature vector n is calculated by the above-described equation (1) and stored in the noise frequency feature vector storage unit 61 in advance.

In this figure, a vector X1 indicates an input frequency feature vector corresponding to the frequency spectrum W2 of the sound signal x1, and a vector X2 indicates an input frequency feature vector corresponding to the frequency spectrum W3 of the sound signal x2.
The frequency spectrum W2 of the sound signal x1, the frequency spectrum W3 of the sound signal x2, the input frequency feature vector X1, and the input frequency feature vector X2 are generated by the frequency feature vector generation unit 231.

In this figure, the inner product value I ₁ indicates the inner product value of the input frequency feature vector X1 and the noise frequency feature vector n, and the inner product value I ₂ indicates the inner product of the input frequency feature vector X2 and the noise frequency feature vector n. The value is shown.

In the present embodiment, the noise estimation unit 233 estimates noise (estimated noise) included in the sound signal based on the inner product value calculated by the inner product calculation unit 232.
Here, the sound signal x2 is the inner product value _{I 2} is larger than the inner product value _{I 1.} This indicates that the direction of the input frequency feature vector X2 and the direction of the noise frequency feature vector n are close. Therefore, it is estimated that most of the sound signal x2 is noise.
In contrast, the sound signal x1 is the inner product value _{I 1} is smaller than the inner product value _{I 2.} This indicates that the direction of the input frequency feature vector X2 is far from the direction of the noise frequency feature vector n. Therefore, it is estimated that the noise amount of the sound signal x1 is smaller than that of the sound signal x2.
That is, the noise estimation unit 233 can appropriately estimate the noise included (superimposed) in the sound signal.

  In the noise estimation apparatus 150 according to the present embodiment, the noise estimation unit 233 estimates the estimated noise based on the above-described noise estimation method, and supplies the estimated noise to the noise reduction processing unit 24 described above. In FIG. 3, the case where the number of frequency bins is two is illustrated in order to explain the concept in the present embodiment. However, the inner product calculation unit 232 actually calculates the inner product of vectors having more elements. Will be calculated. For example, when the number of samples in one frame is 4096 samples, the inner product calculation unit 232 calculates the inner product of vectors having 4096 elements.

  Next, operations of the imaging device 1 and the signal processing device 100 in the present embodiment will be described.

First, the imaging operation of the imaging device 1 will be described.
In the imaging apparatus 1, for example, when receiving an imaging instruction via the operation unit 80, the CPU 90 stores image data obtained via the imaging unit 10 in the storage medium 200. At this time, the CPU 90 controls the lens driving unit 16 and the camera shake preventing unit 18 to drive the zoom lens 14, the VR lens 13, or the AF lens 12.

  When a user captures a moving image, a sound signal is collected by the microphone 21, and the collected sound signal is converted into a sound signal via the A / D conversion unit 22 and the sound signal processing unit 23. Stored in the storage medium 200. In this case, noise may be superimposed on the sound signal collected by the microphone 21 by driving the zoom lens 14, the VR lens 13, or the AF lens 12 or by operating the operation unit 80. The imaging device 1 according to the present embodiment includes a signal processing device 100 including a sound signal processing unit 23, and the signal processing device 100 executes a noise reduction process for reducing noise superimposed on the sound signal.

Next, operations related to noise reduction processing in the signal processing apparatus 100 will be described.
FIG. 4 is a flowchart showing the operation of noise reduction processing in the present embodiment.

In this figure, first, the signal processing apparatus 100 performs Fourier transform on the input sound signal in units of frames (step S101). That is, the frequency feature vector generation unit 231 of the sound signal processing unit 23 converts the sound signal converted into the digital signal by the A / D conversion unit 22 into the frequency spectrum of the sound signal in units of frames. Note that the frequency feature vector generation unit 231 uses, for example, a Hamming window as a window function when performing Fourier transform.
Next, the frequency feature vector generation unit 231 generates an input frequency feature vector (X _k ) based on the frequency spectrum converted into frames (step S102).

Next, the signal processing apparatus 100 calculates the inner product value of the input frequency feature vector and the noise frequency feature vector (step S103). That is, the inner product calculation unit 232 calculates the inner product value of the input frequency feature vector (X _k ) supplied from the frequency feature vector generation unit 231 and the noise frequency feature vector (n) read from the noise frequency feature vector storage unit 61. Is calculated as the noise similarity.

Next, the signal processing apparatus 100 estimates estimated noise (step S104). That is, the noise estimation unit 233 estimates the estimated noise (vector ^to n _k ) included in the sound signal based on the noise similarity (inner product value <X _k , n>) calculated by the inner product calculation unit 232. . The noise estimation unit 233, for example, by equation (2) described above, it calculates the estimated noise (vector ^~ n _k).

Next, the signal processing apparatus 100 subtracts the noise by the estimated noise (step S105). That is, the noise subtraction unit 234 of the noise reduction processing unit 24 includes an input frequency feature vectors (X _k), based on the estimated noise (vector ^{~ n} _k), by subtracting the noise contained in the sound signal, the noise calculating estimated frequency feature vectors of the subtraction purpose sound ^{(~ S} _k). Noise subtraction unit 234, for example, by the above-described formula (3), to calculate an estimated frequency feature vector of the target sound ^(~ _{S k).}

Next, the signal processing apparatus 100 synthesizes a sound signal by performing inverse Fourier transform (step S106). In other words, the inverse transform unit 235, first, back the estimated frequency characteristic vector of the target sound calculated by the noise subtraction unit 234 ^{(~ S} _k) in the frequency spectrum. Then, the inverse conversion unit 235 converts it into a time waveform based on the phase information of the sound signal mixed with noise, and synthesizes the sound signal of the target sound with reduced noise. Further, the inverse transform unit 235 stores the generated sound signal in the storage medium 200 via the communication unit 70 and ends the noise reduction process in one frame.

  Note that the processes in steps S101 to S106 are repeatedly executed until the CPU 90 gives an instruction to end the noise reduction process using a control signal.

  As described above, in the noise estimation device 150, the inner product calculation unit 232 has the noise similarity indicating the degree of similarity between the sound signal and the noise based on the frequency spectrum of the input sound signal and the frequency spectrum of the noise. The degree (inner product value) is calculated. Then, the noise estimation unit 233 estimates the estimated noise included in the sound signal based on the noise similarity calculated by the inner product calculation unit 232.

  Thereby, the noise estimation apparatus 150 can estimate the estimated noise contained in a sound signal appropriately. In addition, the signal processing device 100 reduces noise included in the sound signal based on the estimated noise estimated by the noise estimation device 150 by the noise reduction processing unit 24. For this reason, the noise estimation device 150 and the signal processing device 100, for example, may reduce noise or remain due to excessive noise subtraction or excessive subtraction even when noise whose magnitude is unsteady is reduced. Occurrence can be reduced.

In addition, the noise estimation device 150 and the signal processing device 100 can prevent excessive noise subtraction even when, for example, intermittent noise is reduced, and sound degradation occurs. Can be reduced. Therefore, the noise estimation device 150 and the signal processing device 100 can appropriately reduce noise superimposed on the sound signal.
In addition, the noise estimation device 150 and the signal processing device 100 do not need a plurality of microphones for one-channel sound signal in order to obtain estimated noise of non-stationary noise. Therefore, non-stationary noise can be appropriately reduced with a small number of microphones.

  Further, in the present embodiment, the inner product calculation unit 232 includes an input frequency feature vector generated based on the frequency spectrum of the sound signal and a noise frequency feature vector indicating noise characteristics generated based on the frequency spectrum of noise. The noise similarity is calculated based on the inner product value. Here, the noise frequency feature vector is a vector generated by using the intensity corresponding to each frequency component (each frequency bin) in the frequency spectrum of noise as an element. Further, the input frequency feature vector is a vector generated with the intensity corresponding to each frequency component in the frequency spectrum of the sound signal as an element.

  The inner product value of the input frequency feature vector and the noise frequency feature vector indicates the degree of closeness between the direction of the input frequency feature vector and the direction of the noise frequency feature vector, as shown in FIG. Therefore, the noise estimation device 150 can appropriately estimate the estimated noise based on the inner product value of the input frequency feature vector and the noise frequency feature vector. Therefore, the noise estimation device 150 and the signal processing device 100 can appropriately reduce noise superimposed on the sound signal.

In the present embodiment, the noise frequency feature vector is a unit vector.
As a result, the inner product value calculated by the inner product calculating unit 232 can be prevented from becoming an extremely large value, and the inner product value can be limited.

In the present embodiment, the mode in which the unit vector (n) is used as the noise frequency feature vector when the inner product is calculated by the inner product calculation unit 232 has been described, but the noise frequency feature vector (n ₀ ) that is not normalized is used. Form may be sufficient. In this case, as shown in the equation (4), the noise estimation unit 233 multiplies the noise frequency feature vector by a coefficient α determined according to the noise similarity (for example, inner product value), and calculates the estimated noise. It may be estimated. Here, the coefficient α is, for example, “0.5” for the noise of the AF lens 12 and “0.1” for the noise of the zoom lens 14. ) May be changed according to

In this case, the noise subtraction unit 234, the relational expression shown in equation (5), to calculate an estimated frequency feature vectors ^~ S _k of the target sound.

  Thereby, it is not necessary to normalize the noise frequency feature vector, and the generation process of the noise frequency feature vector can be simplified. Further, since the noise reduction amount can be corrected by the coefficient α as necessary, the noise estimation device 150 and the signal processing device 100 can appropriately reduce noise superimposed on the sound signal.

Next, a method for generating a noise frequency feature vector in the present embodiment will be described.
The noise frequency feature vector is generated in advance by a calibration device, for example, and stored in the noise frequency feature vector storage unit 61 at the time of manufacturing or shipping inspection of the imaging device 1.
The calibration device operates an operation unit (mechanism unit) such as the AF lens 12 in a quiet environment (an environment with low floor noise), and acquires a sound signal of noise collected by the microphone 21 of the imaging device 1. . The calibration device performs a Fourier transform on the noise signal in units of frames to generate a noise frequency spectrum.

Then, the calibration device generates a noise frequency feature vector with the intensity corresponding to each frequency component (each frequency bin) in the frequency spectrum of noise as an element. The noise frequency feature vector may be a normalized vector (n) or a non-normalized vector (n ₀ ) as described above. The calibration apparatus stores the generated noise frequency feature vector in the noise frequency feature vector storage unit 61.
That is, the noise frequency feature vector is generated in advance based on a sound signal obtained when an operation unit (mechanism unit) that generates noise is operated.

  Thereby, since the noise estimation apparatus 150 in this embodiment can obtain an appropriate noise frequency feature vector, it can estimate the noise contained in a sound signal appropriately. Therefore, the noise estimation device 150 and the signal processing device 100 can appropriately reduce noise superimposed on the sound signal.

The calibration device calculates a frequency feature vector in a plurality of frames divided by a predetermined time for a predetermined section of the noise signal of noise, and each frequency component of the noise frequency feature vector has a minimum value thereof, Any one of an average value, an intermediate value, and a maximum value may be applied.
For example, when priority is given to noise reduction, the maximum value in the frequency feature vector in a plurality of frames is applied. For example, when it is desired to suppress the deterioration of the target sound, the minimum value in the frequency feature vector in a plurality of frames is applied. Further, for example, when it is desired to appropriately reduce noise with a good balance between noise reduction and target sound deterioration, an average value or an intermediate value in frequency feature vectors in a plurality of frames is applied.

Next, another example of the noise frequency feature vector generation method in this embodiment will be described.
FIG. 5 is an explanatory diagram illustrating an example of a sound signal on which noise is superimposed in the present embodiment.
In this figure, the horizontal axis indicates time t. FIG. 5A is a graph showing the operation state of the operation unit. When the operation unit is in the H (high) state, the operation unit is operating. When the operation unit is in the L (low) state, the operation is performed. The state where the part stops is shown. In FIG. 5A, a waveform W4 indicates that the operation unit has started operation at time T1.

  Next, FIG. 5B shows an example of a frame into which the sound signal is divided. FIG. 5B shows an example in which a frame is divided from the frame F1 to the frame F7, with a section obtained by shifting a predetermined period by a half period as one frame.

  FIG. 5C is a graph showing the waveform of the sound signal collected by the microphone 21. In FIG. 5C, a waveform W5 indicates a sound signal in a period up to time T1 (period before noise is generated), and a waveform W6 is a period after time T1 (period after noise is generated). A sound signal is shown. As shown in the waveform W6, noise is superimposed on the sound signal during the period after time T1 due to the operation of the operation unit (the driving of the zoom lens 14, the VR lens 13, or the AF lens 12 and the operation of the operation unit 80). Is done.

  In the present embodiment, the calibration device first acquires a sound signal (waveform W5 and waveform W6) as shown in FIG. Then, the calibration device calculates the frequency spectrum (W9) of noise based on the sound signal acquired in this way, as shown in FIG.

  FIG. 6 is an explanatory diagram illustrating an example of a method for generating a noise frequency spectrum in the present embodiment. In this figure, a frequency spectrum W7 indicates a frequency spectrum before noise generation (before time T1, for example, frame F1), and a frequency spectrum W8 indicates a frequency spectrum after noise generation (after time T1, for example, frame F4). Show. The frequency spectrum W7 indicates a background noise (floor noise) component included as a stationary noise in the sound signal. The frequency spectrum W9 indicates the frequency spectrum of noise obtained by subtracting the frequency spectrum W7 from the frequency spectrum W8 (subtracting the frame F1 from the frame F4).

  That is, the calibration apparatus calculates a noise frequency spectrum as shown in FIG. 6, and generates a noise frequency feature vector based on the calculated noise frequency spectrum. That is, the noise frequency feature vector is generated by subtracting the background noise component included as stationary noise in the sound signal from the sound signal when the mechanism unit is operated. The calibration apparatus stores the generated noise frequency feature vector in the noise frequency feature vector storage unit 61.

  As a result, the background noise (floor noise) component contained as stationary noise in the sound signal from the noise frequency feature vector is reduced, so that the inner product of the input frequency feature vector and the noise frequency feature vector by the background noise (floor noise). It is possible to prevent the value from becoming a large value. Therefore, the noise estimation apparatus 150 in the present embodiment can appropriately estimate noise included in the sound signal. Therefore, the noise estimation device 150 and the signal processing device 100 can appropriately reduce noise superimposed on the sound signal.

  Even when the noise frequency feature vector is subtracted from the background noise described above, a method that uses any one of the minimum value, average value, intermediate value, and maximum value of the frequency feature vector in the plurality of frames described above is applied. May be.

According to the embodiment of the present invention, the imaging device 1 includes the signal processing device 100 described above.
Thereby, the imaging device 1 can expect the effect similar to the signal processing apparatus 100, and can reduce the noise superimposed on the sound signal appropriately.

Further, according to the embodiment of the present invention, the inner product calculation unit 232 is added to the computer as the noise estimation device 150 based on the frequency spectrum of the input sound signal and the frequency spectrum of the noise. The noise calculation unit (step S103) for calculating the noise similarity indicating the degree of similarity with the noise estimation unit 233 estimates the estimated noise included in the sound signal based on the noise similarity calculated by the calculation procedure. This is a program for executing the noise estimation procedure (step S104).
Thereby, the program can expect the same effect as the noise estimation apparatus 150, and can appropriately reduce the noise superimposed on the sound signal.

The present invention is not limited to the above embodiments, and can be modified without departing from the spirit of the present invention.
In each of the embodiments described above, the noise estimation unit 233 has described a mode of estimating the estimated noise based on the inner product value of the input frequency feature vector and the noise frequency feature vector calculated as the noise similarity. It is not limited. For example, the calculation unit that calculates the noise similarity calculates the noise similarity based on the correlation coefficient between the frequency spectrum of the sound signal and the frequency spectrum of the noise, and the noise estimation unit 233 calculates the correlation coefficient. A form in which estimated noise is estimated based on this may be used. In this case, the calculation unit that calculates the noise similarity calculates a correlation coefficient by each set corresponding to each frequency component (each frequency bin) of the frequency spectrum of the sound signal and the frequency spectrum of the noise.

Thereby, the noise estimation apparatus 150 and the signal processing apparatus 100 can reduce appropriately the noise superimposed on the sound signal similarly to the form which estimates estimated noise based on the inner product value.
The noise similarity may be calculated based on, for example, the Mahalanobis distance, or may be in another form.

  Further, in each of the embodiments described above, the form in which the absolute value is used as the intensity of the noise frequency spectrum and the intensity of the frequency spectrum of the sound signal has been described. However, a form using complex numbers and power including an imaginary part may be used. .

  In each of the above embodiments, the signal processing apparatus 100 includes a detection unit that detects the generation timing of noise in the operation unit (mechanism unit), and reduces noise based on the generation timing detected by the detection unit. It may be determined whether or not to perform processing. That is, the signal processing apparatus 100 may be configured to execute a process of reducing noise when it is determined that noise is generated based on the generation timing detected by the detection unit.

  Also, the noise estimation device 150 selects and uses one or more of the noise frequency feature vectors stored in the noise frequency feature vector storage unit 61 based on the generation timing detected by the detection unit. Form may be sufficient. For example, the noise estimation device 150 detects that any one of the mechanical sounds of the zoom lens 14, the VR lens 13, and the AF lens 12 is operating by the detection unit, and zooms in according to the detection result. A noise frequency feature vector corresponding to any one of the lens 14, the VR lens 13, and the AF lens 12 may be selected.

In the above embodiment, the noise subtraction unit 234, based on the estimated noise estimated by the noise estimation unit 233 has been described a mode of estimating the estimated frequency characteristic vector ^~ S _k of the target sound, furthermore, weighted The calculation may be performed by adding a coefficient. For example, the SNR (signal-noise ratio), which is the ratio of the frequency spectrum of the sound signal before noise generation to the estimated noise, the tone degree, and the like are calculated, and the estimated noise amount is calculated by multiplying the weighted coefficient based on the calculated value. It may be corrected.

In each of the above-described embodiments, the noise frequency feature vector is generated by the calibration device and stored in the noise frequency feature vector storage unit 61. However, the imaging device 1, the signal processing device 100, or the noise estimation device is described. 150 may include a generation unit that generates a noise frequency feature vector. Alternatively, the calibration apparatus may store the noise frequency feature vector that has already been generated in the noise frequency feature vector storage unit 61.
Further, in each of the embodiments described above, the noise frequency feature vector storage unit 61 has been described as storing noise frequency feature vectors. However, the noise frequency feature vector storage unit 61 stores a frequency spectrum of noise and performs noise estimation. The apparatus 150 may generate the noise frequency feature vector. For example, the inner product calculation unit 232 may generate a noise frequency feature vector based on the frequency spectrum of noise.

Further, in each of the above embodiments, the mode in which the signal processing device 100 is applied to the imaging device 1 has been described. However, the present invention is not limited to this. For example, the present invention may be applied to a recording device such as a recorder or a device in which sound generated by key operations such as a telephone is superimposed as a noise signal as a target sound as noise.
Further, in each of the embodiments described above, the form using FFT as the Fourier transform has been described. However, a form using DFT (Discrete Fourier Transform) may be used, or a form using another method may be used.

  The noise estimation device 150 and the signal processing device 100 described above have a computer system therein. The processing steps of the noise estimation device 150 and the signal processing device 100 described above are stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer reading and executing the program. Is called. Here, the computer-readable recording medium refers to a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, and the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 24 ... Noise reduction process part, 61 ... Noise frequency feature vector memory | storage part, 100 ... Signal processing apparatus, 150 ... Noise estimation apparatus, 231 ... Frequency feature vector generation part, 232 ... Inner product calculation part, 233 ... Noise estimation unit

Claims (11)

A calculation unit that calculates a noise similarity indicating a degree of similarity between the sound signal and the noise based on the frequency spectrum of the input sound signal and the frequency spectrum of the noise;
A noise estimation device comprising: a noise estimation unit that estimates estimated noise included in the sound signal based on the noise similarity calculated by the calculation unit. The calculation unit includes:
Based on an inner product value of an input frequency feature vector generated based on the frequency spectrum of the sound signal and a noise frequency feature vector indicating the feature of the noise generated based on the frequency spectrum of the noise, the noise similarity Calculate the degree,
The noise frequency feature vector is a vector generated by using the intensity corresponding to each frequency component in the frequency spectrum of the noise as an element,
The noise estimation apparatus according to claim 1, wherein the input frequency feature vector is a vector generated using an intensity corresponding to each frequency component in a frequency spectrum of the sound signal as an element. The noise estimation apparatus according to claim 2, wherein the noise frequency feature vector is a unit vector. The noise estimation unit
The noise estimation according to any one of claims 2 to 3, wherein the estimated noise is estimated by multiplying the noise frequency feature vector by a coefficient determined in accordance with the noise similarity. apparatus. A storage unit in which the noise frequency feature vector is stored in advance;
The frequency feature vector generation part which converts the sound signal into a frequency spectrum, and generates the input frequency feature vector based on the converted frequency spectrum of the sound signal. The noise estimation apparatus according to any one of the above. 6. The noise frequency feature vector is generated in advance based on a sound signal obtained when a mechanism unit that generates the noise is operated. The noise estimation apparatus described in 1. The noise frequency feature vector is generated by subtracting, from the sound signal, a background noise component included as stationary noise in the sound signal when the mechanism unit is operated. The noise estimation apparatus described in 1. The calculation unit includes:
The noise estimation device according to claim 1, wherein the noise similarity is calculated based on a correlation coefficient between a frequency spectrum of the sound signal and a frequency spectrum of the noise. The noise estimation device according to any one of claims 1 to 8,
A noise reduction processing unit that reduces noise included in the sound signal based on the estimated noise estimated by the noise estimation device;
A signal processing apparatus comprising:   An image pickup apparatus comprising the signal processing apparatus according to claim 9. In the computer as a noise estimation device,
A calculation procedure for calculating a noise similarity indicating a degree of similarity between the sound signal and the noise based on the frequency spectrum of the input sound signal and the frequency spectrum of the noise;
A noise estimation unit for executing a noise estimation procedure for estimating an estimated noise included in the sound signal based on the noise similarity calculated by the calculation procedure.

Images