JP2012118317A - Signal processor, imaging apparatus and signal processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique which enables acquisition of a sound signal with inconspicuous non-steady sound without deterioration in the sound quality of the sound signal.SOLUTION: A signal processor includes: a storage unit for storing information on drive noise generated from an imaging apparatus; an operation unit for calculating similarity showing the degree of similarity between recorded sound and the drive noise by using the information on the drive noise; and a control unit for determining whether the sound and the drive noise are similar or not based on the similarity and instructing control on drive operation with the drive noise according to the determination result.

Description

  The present invention relates to a signal processing device, an imaging device, and a signal processing program.

  2. Description of the Related Art Conventionally, in audio recording and moving image capturing by a digital camera, a digital video camera, or the like, not only ambient sounds but also background noise and driving sounds emitted by the digital camera and digital video camera itself are recorded. Therefore, various techniques for removing only such background noise and driving sound and recording only necessary sound have been developed.

  For example, Non-Patent Document 1 discloses a technique for removing background noise superimposed on an audio signal using a spectral subtraction method.

S. F. Boll, 'Suppression of Acoustic Noise in Speech Using Spectral Subtraction', IEEE TRANSACTIONS ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, Vol.ASSP-27, No. 2, pp.113-120, April 1979

  However, with the conventional technology such as Patent Document 1, it is very difficult to remove non-stationary sounds such as driving sounds as well as background noise without deteriorating the sound quality of the audio signal.

  SUMMARY OF THE INVENTION In view of the above-described problems of the prior art, an object of the present invention is to provide a technique that can acquire an audio signal in which unsteady sound is not noticeable without deteriorating the sound quality of the audio signal.

  In order to solve the above-described problem, the signal processing device of the present invention has a storage unit that stores information on the driving sound emitted by the imaging device, and a degree of similarity indicating the degree of similarity between the recorded sound and the driving sound. A calculation unit that calculates using information, and a control unit that determines whether the sound and the driving sound are similar based on the degree of similarity, and that controls the driving operation with the driving sound according to the determination result, Prepare.

  Further, the control unit may instruct the control to perform the driving operation when it is determined that the sound and the driving sound are similar.

  In addition, when it is determined that the sound and the driving sound are not similar, the control unit may instruct the control to cancel the driving operation.

  The calculation unit further includes an analysis unit that generates first feature value data including a plurality of feature values acquired from the frequency characteristics of the sound, and the drive sound information includes a plurality of information acquired from the frequency characteristics of the drive sound. It is 2nd feature-value data which consist of feature-values, Comprising: A calculating part may calculate a similarity based on the calculation of the inner product using 1st feature-value data and 2nd feature-value data.

  In addition, a normalization unit that normalizes the first feature amount data and the second feature amount data is provided, and the arithmetic unit uses the first feature amount data and the second feature amount data that have been normalized to perform an inner product. May be performed.

  The driving operation may be a focus operation or a zoom operation by the imaging device, or a dial operation or a button operation by the user.

  Moreover, you may provide the display part which displays the similarity or the determination result by a control part.

  An imaging apparatus of the present invention includes an imaging unit that captures a subject image and generates an image, a microphone that outputs an external sound as an audio signal, and the signal processing apparatus of the present invention.

  Moreover, you may provide the display part which displays an image with the similarity or the determination result by a control part.

  The signal processing program of the present invention causes a computer to function as the signal processing device of the present invention.

  According to the present invention, it is possible to acquire an audio signal in which unsteady sound is not noticeable without deteriorating the sound quality of the audio signal.

The block diagram which shows an example of a structure of the digital camera 1 which concerns on one Embodiment. Flow chart showing an operation example of the digital camera 1 The figure which shows an example of the display of the monitor 17 The figure which shows the other example of a display of the monitor 17

  FIG. 1 is a block diagram illustrating an example of a configuration of a digital camera 1 according to an embodiment.

  The digital camera 1 of the present embodiment includes an imaging optical system 11, an imaging device 12, a DFE 13, a CPU 14, a memory 15, an operation unit 16, a monitor 17, a media I / F 18, and a microphone 20. Here, the DFE 13, the memory 15, the operation unit 16, the monitor 17, the media I / F 18, and the microphone 20 are each connected to the CPU 14.

  The image pickup device 12 is a device that picks up an image of an object formed by a light beam that has passed through the image pickup optical system 11. The output of the image sensor 12 is connected to the DFE 13. Note that the image sensor 12 of the present embodiment may be a progressive scanning solid-state image sensor (CCD or the like) or an XY address type solid-state image sensor (CMOS or the like).

  A plurality of light receiving elements are arranged in a matrix on the light receiving surface of the imaging element 12. In each light receiving element of the image sensor 12, red (R), green (G), and blue (B) color filters are arranged according to a known Bayer array. Therefore, each light receiving element of the imaging element 12 outputs an image signal corresponding to each color by color separation in the color filter. Thereby, the image sensor 12 can acquire a color image at the time of imaging. The image sensor 12 outputs an image signal to all the light receiving elements to pick up a still image or the like, and outputs an image signal to a part of the thinned light receiving elements to output a low resolution image (through image) for checking a moving image or a composition. And so on.

  The DFE 13 is a digital front-end circuit that performs signal processing such as A / D conversion of image signals input from the image sensor 12 and correction of defective pixels. In this embodiment, the DFE 13 constitutes an image pickup unit together with the image pickup device 12 and outputs an image signal input from the image pickup device 12 to the CPU 14 as image data.

  The CPU 14 is a processor that comprehensively controls each unit of the digital camera 1. For example, the CPU 14 performs autofocus (AF) control by known contrast detection, known automatic exposure (AE) calculation, and the like based on the output of the image sensor 12. Further, the CPU 14 performs digital processing on the image data from the DEF 13. As an example, digital processing includes interpolation processing, white balance processing, gradation conversion processing, contour enhancement processing, color conversion processing, and the like. Further, the CPU 14 operates as a spectrum analysis unit 21, a calculation unit 22, and a normalization unit 23 by executing the program.

The spectrum analysis unit 21 performs spectrum analysis on a time-series signal of surrounding sounds acquired by the microphone 20 described later when a digital camera 1 captures a moving image or the like. The spectrum analysis of the present embodiment is performed using a known method such as fast Fourier transform, Z transform, or discrete cosine transform. The spectrum analysis unit 21 generates a plurality of feature amounts of the frequency spectrum of the speech signal obtained from the spectrum analysis, that is, a feature vector V _S having the vector components of the amplitudes of the plurality of frequency components.

Further, in the present embodiment, the CPU 14 responds to the operation of the operation unit 16 by the user (for example, a half-press operation of the release button) in a silent state without surrounding sound before the digital camera 1 performs imaging. Then, a time series signal of the driving sound emitted from the digital camera 1 is acquired via the microphone 20. The spectrum analysis unit 21 performs spectrum analysis on the acquired drive sound signal to generate a feature vector V _N. The spectrum analysis unit 21 records the feature vector V _N in the memory 15. Note that the drive sound is an unsteady sound that is emitted in association with AF control or the zooming operation of the imaging optical system 11, or is used for manual focus operation or operation of the operation unit 16 (for example, button operation or dial operation) by the user. It is an unsteady sound that is accompanied. The feature vector V _N of the driving sound signals may be used several times in other imaging Once prompted, it may be updated by acquiring the driving sound each time for imaging.

The calculation unit 22 obtains an inner product value using the unit feature vectors e _S and e _N of the feature vectors V _S and V _N normalized by the normalization unit 23. The inner product value of the present embodiment is a similarity indicating the degree of similarity between the acquired audio signal and the driving sound signal of the digital camera 1. The inner product value (similarity) is a value from 0 to 1, and a larger value indicates that the audio signal and the drive sound signal are similar. That is, when the inner product value = 0, the sound signal is completely different from the drive sound signal, and when the inner product value = 1, the sound signal = the drive sound signal.

Memory 15, such as with various programs and image files to be executed by the CPU 14, a nonvolatile flash memory for storing a feature vector V _N of the drive sound signal.

  The operation unit 16 receives, for example, a recording mode switching input, an instruction to capture a still image or a moving image that accompanies recording in a storage medium 19 described later, and the like from the user.

  The monitor 17 displays various images under the control of the CPU 14. For example, after capturing a moving image, the monitor 17 reproduces and displays the captured moving image in response to a control instruction from the CPU 14. In addition, the monitor 17 of the present embodiment displays a determination result of whether or not the audio signal and the driving sound signal by the CPU 14 are similar based on the similarity. As the monitor 17, a monitor such as a liquid crystal monitor, an electronic viewfinder, or the like can be appropriately selected and used.

  A non-volatile storage medium 19 can be detachably connected to the media I / F 18. The media I / F 18 executes data writing / reading with respect to the storage medium 19. The storage medium 19 includes a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is illustrated as an example of the storage medium 19.

  The microphone 20 receives not only the surrounding sound but also the driving sound emitted by the digital camera 1 and outputs the sound signal and the electric signal of the driving sound signal. The microphone 20 is a speaker that outputs a recorded audio signal as a sound wave when reproducing a moving image or the like.

  Next, an operation example of the digital camera 1 in this embodiment will be described with reference to the flowchart of FIG. In the following description, the driving sound signal is a driving sound during the AF control operation.

  Specifically, the CPU 14 accepts a moving image capturing instruction from the user (for example, full pressing operation of a release button included in the operation unit 16). The CPU 14 outputs an image signal from a part of the light receiving elements thinned out by the imaging element 12 and starts capturing a moving image of the subject. As shown in FIG. 3, the CPU 14 displays a through image 40 showing the composition of the moving image on the monitor 17. At the same time, the CPU 14 causes the microphone 20 to acquire the subject and surrounding sounds and output a sound signal. CPU14 starts the process of step S101-step S105.

Step S101: The spectrum analysis unit 21 applies a predetermined window function to the audio signal output from the microphone 20 at predetermined time intervals (for example, 10 milliseconds), and performs fast Fourier transform, Z transform, or discrete cosine transform. The spectrum is analyzed using a known method such as the above. The spectrum analysis unit 21 generates a feature vector V _S from the frequency spectrum of the speech signal obtained by the spectrum analysis.

Step S102: calculating section 22 reads the feature vector _{V N} of the drive sound signal from the memory 15. The normalizing unit 24 normalizes the feature vector V _S of the audio signal and the feature vector V _N of the driving sound signal to obtain unit vectors e _S and e _N.

Step S103: The calculation unit 22 obtains an inner product value e _S × e _N of the unit vectors e _S and e _N as the similarity.

Step S104: The CPU 14 determines whether or not the sound signal and the drive sound signal are similar based on the inner product value e _S × e _N obtained by the calculation unit 22. The CPU 14 displays the determination result on the monitor 17 and notifies the user (FIG. 3).

FIG. 3 shows an example of the determination result displayed on the monitor 17. In the present embodiment, together with the through image 40, the similarity of the determination result of the audio signal to the driving sound signal is displayed on the indicator 30 and notified to the user. FIG. 3A shows a case where an image is picked up on a plateau as an example in which the similarity between the audio signal and the driving sound signal of the AF control operation is small, and FIG. As an example of a large similarity value, a case where an image is taken on the side of a road with a large amount of traffic is shown. In addition, as shown in FIG. 4, you may display the determination result for every sound source of a drive sound like the indicator 35. FIG. The indicator 35 shows the result of the similarity degree with respect to each sound signal of the driving sound of the manual focus operation (MF) and the zooming operation (ZM) together with the driving sound of the AF control operation. Further, instead of the indicator 30, characters, colors, symbols, or the like may be displayed. When displaying in characters, colors, symbols, or the like, a threshold value is stored in the memory 15 in advance, and the CPU 14 determines whether the sound signal and the drive sound signal are similar based on the inner product value e _S × e _N and the threshold value. It is preferable to determine whether or not.

  When the similarity value is small as shown in FIG. 3A and the audio signal and the drive sound signal are not similar, if the digital camera 1 performs the AF control operation during moving image shooting, the superimposed drive during reproduction of the moving image is performed. The sound signal is noticeable. Therefore, the CPU 14 displays the determination result on the monitor 17 with the indicator 30, so that the user determines whether to postpone the AF control operation by the digital camera 1 based on the display of the indicator 30. As a result, it is possible to acquire an audio signal on which the driving sound signal by the AF control operation is not superimposed.

  On the other hand, when the similarity as shown in FIG. 3B is large and the audio signal and the drive sound signal are similar, even when the digital camera 1 performs the AF control operation when capturing the moving image, The drive sound signal does not stand out. Accordingly, the CPU 14 displays the determination result on the monitor 17 with the indicator 30, so that the user determines whether or not to perform the AF control operation by the digital camera 1 based on the indicator 30. Thereby, it is possible to acquire an audio signal in which the driving sound signal by the AF control operation is not conspicuous.

  Step S105: The CPU 14 temporarily records the image data and the acquired audio signal in the memory 15 as moving image data in accordance with a predetermined moving image format such as Audio Video Interleave (AVI) or Motion JPEG. The CPU 14 determines whether or not the user has fully released the release button. When the CPU 14 determines that the release button has been fully pressed (YES side), the CPU 14 generates a moving image file of the moving image data temporarily recorded in the memory 15 and records it in the memory 15 or the storage medium 19. The CPU 14 ends a series of processes. On the other hand, if the CPU 14 determines that the release button has not been fully pressed, the process proceeds to step S101 (NO side). CPU14 performs the process of step S101-step S104 until the full press of a release button is cancelled | released.

Thus, in the present embodiment, the sound signal is driven based on the inner product value (similarity) between the acquired feature vector V _S of the sound signal and the feature vector V _N of the drive sound signal stored in the memory 15. By determining whether or not it is similar to the sound signal and restricting the operation of the digital camera 1, it is possible to obtain a sound signal in which the superimposition of the drive sound signal is not noticeable without deteriorating the sound quality of the sound signal.
<< Additional items of embodiment >>
(1) The signal processing apparatus of the present invention is not limited to the example of the digital camera 1 of the above embodiment. For example, a computer may function as the signal processing apparatus of the present invention.

(2) In the above embodiment, the inner product value e _S × e _N is used as the similarity, but a correlation value between the unit vectors e _S and e _N may be used.

  (3) In the above-described embodiment, an example in which the CPU 14 implements each process of the spectrum analysis unit 21, the calculation unit 22, and the normalization unit 23 of the digital camera 1 by software has been described. Of course, each processing may be realized by hardware.

  (4) In the above embodiment, when the similarity between the audio signal and the drive sound signal as shown in FIG. 3A is small, the CPU 14 displays the determination result on the monitor 17 with the indicator 30 to drive the user. Although the operation of the digital camera 1 with sound is limited, the following may be performed. For example, in general, the frequency spectrum of an audio signal and a drive sound signal has few overlapping portions in the frequency space. Therefore, even if an audio signal on which a drive sound signal accompanying the operation of the digital camera 1 is acquired, it is possible to easily remove the drive sound signal superimposed from the audio signal without degrading the sound quality in subsequent processing.

  (5) In the above embodiment, the determination process is performed to determine whether or not the audio signal is similar to the drive sound signal at the time of moving image capturing. A determination process may be performed.

(6) In the above embodiment, the inner product value e _S × e _N obtained by the calculation unit 22 in step S103 is used only for the similar determination between the audio signal and the driving sound signal, but the inner product value e _S × e _N may be added to the moving image file as similarity data. Thereby, at the time of reproduction or the like, it is possible to perform a drive sound signal removal process or the like based on the similarity data, and avoid deterioration of sound quality.

(7) In the above embodiment, one feature vector V _N is obtained from one drive sound signal, but a plurality of feature vectors V _N may be obtained from one drive sound signal. For example, in the case of driving sound of an AF control operation, generally, a period until the imaging optical system 11 is brought into a steady driving operation from a stop state by a lens driving unit (not shown), a steady driving operation section, a steady driving. The driving sound signals in the section from the operation to the stop state have different frequency spectra. Therefore, the spectrum analysis unit 21 performs spectrum analysis on the driving sound signal in each section, and generates a feature vector V _N in each section. The spectrum analysis unit 21 records these feature vectors V _N in the memory 15. As a result, the CPU 14 determines whether or not the sound signal is similar to the drive sound signal based on the inner product value of the feature vector V _{S of the} sound signal and the feature vector V _{N of} each section obtained by the calculation unit 22. Can be done accurately.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 1 ... Digital camera, 11 ... Imaging optical system, 12 ... Imaging device, 13 ... DEF, 14 ... CPU, 15 ... Memory, 16 ... Operation part, 17 ... Monitor, 18 ... Media I / F, 19 ... Storage medium, 20 ... Microphone, 21 ... Spectrum analysis unit, 22 ... Calculation unit, 23 ... Normalization unit

Claims (10)

A storage unit for storing information on driving sound emitted by the imaging device;
A calculation unit that calculates a degree of similarity indicating the degree of similarity between the recorded sound and the driving sound using the information of the driving sound;
A control unit that determines whether the sound and the driving sound are similar based on the similarity, and that controls the driving operation with the driving sound according to the determination result;
A signal processing apparatus comprising: The signal processing device according to claim 1,
The signal processing device according to claim 1, wherein the control unit issues a control instruction to perform the driving operation when it is determined that the sound and the driving sound are similar. The signal processing device according to claim 1 or 2,
The signal processing apparatus according to claim 1, wherein when the control unit determines that the sound and the drive sound are not similar, the control unit issues a control instruction when the drive operation is delayed. The signal processing apparatus according to any one of claims 1 to 3,
The computing unit is
An analysis unit that generates first feature amount data including a plurality of feature amounts acquired from the frequency characteristics of the voice;
The drive sound information is second feature value data including a plurality of feature values acquired from the frequency characteristics of the drive sound,
The signal processing device, wherein the calculation unit calculates the similarity based on an inner product calculation using the first feature value data and the second feature value data. The signal processing device according to claim 4,
A normalization unit that normalizes the first feature amount data and the second feature amount data;
The computing unit is
The signal processing device, wherein the inner product is calculated using the normalized first feature value data and the second feature value data. The signal processing device according to any one of claims 1 to 5,
The signal processing device, wherein the driving operation is a focus operation or a zoom operation by the imaging device, or a dial operation or a button operation by a user. The signal processing device according to any one of claims 1 to 6,
A signal processing apparatus comprising: a display unit configured to display the similarity or the determination result by the control unit. An imaging unit that captures a subject image and generates an image;
A microphone that outputs external audio as an audio signal;
A signal processing device according to any one of claims 1 to 6,
An imaging apparatus comprising: The imaging apparatus according to claim 7,
An image pickup apparatus comprising: a display unit that displays the image together with the similarity or the determination result by the control unit.   A signal processing program for causing a computer to function as the signal processing device according to any one of claims 1 to 7.

Images