JP2011055386A - Audio signal processor, and electronic apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To subject an audio signal suitable for a type of a sound source to slow reproduction in slow reproduction of a video. <P>SOLUTION: An object audio signal for α seconds is collected when imaging an object moving image for α seconds at 600 fps; and, when the object moving image is reproduced at 60 fps (that is, when slow reproduction is performed using (10×α) seconds), an extended audio signal obtained by temporally extending an object audio signal by a factor of 10 is reproduced by being synchronized with reproduction video. When an audio signal of human voice (cheer such as "wahooo") and an audio signal of impulse sound (sound such as "whack") are included in the object audio signal, they are separated and extracted from the object audio signal, a pitch-keeping extension process of extending signal length without changing the pitch is executed to the former, and an echo process of performing repetitive reproduction while being associated with gradual reduction of volume is performed to the latter. A synthetic signal of the audio signal after the processes is reproduced as an extended audio signal along with slow reproduction video. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an acoustic signal processing apparatus that performs signal processing on an acoustic signal. The present invention also relates to an electronic apparatus such as a recording apparatus or a reproducing apparatus using such an acoustic signal processing apparatus.

  With the recent development of imaging technology, imaging apparatuses capable of capturing and recording video at a higher speed than usual have been put into practical use. Such a high-speed shooting function has been conventionally installed only in a special-purpose imaging device, but recently it is also installed in a consumer imaging device.

  With this type of imaging apparatus, it is possible to shoot moving images in the normal shooting mode or the high-speed shooting mode. In the normal shooting mode, as in general moving image shooting, 60 frames or 30 frames of video are shot and recorded per second. That is, moving images are shot at a frame rate of 60 fps (frame per second) or 30 fps. When a moving image recorded in the normal shooting mode is played back at the same frame rate as that at the time of shooting (that is, 60 fps or 30 fps), a playback video at a normal speed is obtained (see FIG. 5).

  On the other hand, in the high-speed shooting mode, moving images are shot at a high-speed frame rate of 300 fps or 600 fps. When a moving image shot in this high-speed shooting mode is played back at a normal frame rate of 60 fps, smooth slow playback at 1/5 times speed or 1/10 times speed can be realized (see FIG. 6).

  For example, if a moving image is shot for 1 second at a frame rate of 600 fps, a moving image consisting of 600 frames is recorded. If this moving image is played back at a frame rate of 60 fps, the playback takes 10 seconds. That is, a recorded moving image for one second is played back slowly (1/10 speed slow playback) over 10 seconds.

  An imaging apparatus capable of slow reproduction based on high-speed shooting has been put into practical use, but in reality, no acoustic signal is recorded in the high-speed shooting mode. When shooting a moving image for 1 second, an acoustic signal for 1 second is picked up and recorded, and the recorded acoustic signal for 1 second is tried to be played slowly in synchronization with the slow-playing moving image for 10 seconds. This is because a sound that is extended by changing the pitch of the acoustic signal is reproduced.

  On the other hand, technologies relating to slow reproduction of acoustic signals are disclosed in Patent Documents 1 to 3 below. In any of the methods disclosed in these patent documents, an expansion process is performed on an acoustic signal in accordance with a recording or reproduction frame rate. The expansion process related to an acoustic signal refers to a process of increasing the signal length of the acoustic signal by stretching the acoustic signal to be expanded in the time direction. The signal length of the acoustic signal refers to the time length of the section where the acoustic signal exists.

  As a general stretching method, a method of stretching an acoustic signal while maintaining the pitch (in other words, a method in which the pitch is not changed before and after the stretching process) is known, and without changing the pitch of the voice. It is applied to speech speed conversion technology that increases or decreases the speech speed. However, it is not desirable to simply apply this method to slow video playback. The method of extending the sound signal while maintaining the pitch is basically a method suitable for extending a human voice. When the sound signal recorded together with the moving image is, for example, a music sound signal, the extension method is used. This is because when applied, a sound with a sense of incongruity is reproduced. The same problem can occur when the acoustic signal recorded with the moving image is from a voice source other than human voice and music.

No. 2007-29832 (WO2007 / 029832) JP 2001-298710 A JP 2008-219857 A

  SUMMARY An advantage of some aspects of the invention is that it provides an audio signal processing device and an electronic apparatus that can generate an audio signal suitable for slow reproduction of video.

  The acoustic signal processing device according to the present invention is an output acoustic signal having a signal length longer than the input acoustic signal from an input acoustic signal picked up when the target moving image is captured at the first frame rate. And an output sound signal generating unit that generates the target moving image when the target moving image is reproduced at a second frame rate smaller than the first frame rate. It is an acoustic signal to be reproduced as sound together with an image, and the output acoustic signal generation unit generates the output acoustic signal from the input acoustic signal according to the type of sound source of the input acoustic signal.

  This makes it possible to generate an audio signal for slow playback of a video adapted to the type of sound source.

  Specifically, for example, the output acoustic signal generation unit includes a sound source type analysis unit that analyzes a type of a sound source of the input sound signal based on the input sound signal, and is analyzed by the sound source type analysis unit, The output sound signal is generated from the input sound signal according to the type of the sound source of the input sound signal.

  Further, for example, the sound source type analyzing unit determines whether or not a human voice is included in the sound source of the input sound signal based on the input sound signal, and the output sound signal generating unit is configured to output the input sound signal. The method for generating the output sound signal from the input sound signal is changed according to whether or not a human voice is included in the sound source of the signal.

  More specifically, for example, when the input acoustic signal includes acoustic signals from a plurality of different sound sources, the output acoustic signal generation unit uses the sound source type analysis unit to generate the plurality of sound sources. After analyzing the sound source type of each separated sound signal while individually extracting the sound signal from the input sound signal as a plurality of separated sound signals, the sound source type of each separated sound signal for each separated sound signal The output sound signal is generated by synthesizing the plurality of separated sound signals after performing a corresponding expansion process.

  Thereby, the expansion process suitable for the kind of sound source can be performed for each sound signal from a plurality of sound sources that can be included in the input sound signal.

  Further, for example, the output sound signal generation unit generates the output sound signal from the input sound signal based not only on the analysis result by the sound source type analysis unit but also on the analysis result on the video signal of the target moving image.

  As a result, it is possible to generate and reproduce an audio signal applied to the video content.

  The electrical device according to the present invention is an electronic device including the audio signal processing, and generates the output acoustic signal from the input acoustic signal when the target moving image is captured at a first frame rate. When the output sound signal is recorded on a recording medium, or the input sound signal is recorded on the recording medium and the target moving image is reproduced at the second frame rate, the recorded input is recorded. The output sound signal is generated from the sound signal, and the output sound signal is reproduced together with the target moving image.

  ADVANTAGE OF THE INVENTION According to this invention, the audio signal processing apparatus and electronic device which can produce | generate the audio signal suitable for slow reproduction | regeneration of an image | video can be provided.

  The significance or effect of the present invention will become more apparent from the following description of embodiments. However, the following embodiment is merely one embodiment of the present invention, and the meaning of the term of the present invention or each constituent element is not limited to that described in the following embodiment. .

1 is a block diagram illustrating an overall configuration of an imaging apparatus according to a first embodiment of the present invention. It is an internal block diagram of the operation part of FIG. It is an internal block diagram of the microphone part of FIG. It is an external appearance perspective view of an imaging device provided with two microphones. It is a reproduction image figure of the moving image image | photographed in normal imaging | photography mode concerning 1st Embodiment of this invention. FIG. 4 is a reproduction image diagram of a moving image shot in a high-speed shooting mode according to the first embodiment of the present invention. It is a block diagram of the site | part which concerns on 1st Embodiment of this invention and is related to the production | generation of an expansion | extension acoustic signal. It is a figure which shows the time relationship between the object moving image at the time of reproduction | regeneration, and an expansion | extension sound signal. It is a figure which shows the reference | standard block and evaluation block which were set with respect to the object acoustic signal in a specific area. It is the figure which showed a mode that the autocorrelation value between a reference | standard block and an evaluation block took a local maximum periodically with respect to the change of the positional difference (p) between both blocks. It is a figure which shows the example of a relationship between a target acoustic signal and an expansion | extension acoustic signal. It is a figure which shows a mode that an expansion | extension acoustic signal is produced | generated from the object acoustic signal through the production | generation of a separated acoustic signal. It is an image figure of the simple expansion | extension process which is a kind of expansion | extension process. It is an image figure of the pitch maintenance expansion | extension process which is a kind of expansion | extension process. It is an image figure of the echo process which is a kind of expansion process. FIG. 5 is an image diagram (a) of normal reproduction of a target sound signal and a target moving image and an image diagram (b) of slow reproduction of the target moving image accompanied by reproduction of the expanded sound signal, according to the first expansion example according to the present invention. . It is a figure which shows a mode that all the sections of a target sound signal are divided | segmented into three areas in the 1st expansion | extension example according to this invention. It is a graph which shows the frequency spectrum (a) of the object acoustic signal containing a cheer and a striking sound, and the Fourier transform (b) of this frequency spectrum concerning the 1st expansion specific example which concerns on this invention. FIG. 5 is an image diagram (a) of normal reproduction of a target sound signal and a target moving image and an image diagram (b) of slow reproduction of the target moving image accompanied by reproduction of the expanded sound signal according to a second expansion example according to the present invention. . It is a figure for demonstrating the positional relationship of two microphones and a sound source concerning the 2nd expansion specific example which concerns on this invention. FIG. 6 is an image diagram (a) of normal reproduction of a target sound signal and a target moving image, and an image diagram (b) of slow reproduction of the target moving image accompanied by reproduction of the expanded sound signal, according to a third expansion example according to the present invention. . It is a figure which shows a mode that all the area | regions of an object acoustic signal are divided | segmented into three area according to the 3rd expansion specific example which concerns on this invention. It is a graph which shows the frequency spectrum of the acoustic signal by a goal utterance, a cheer, and BGM concerning the 3rd expansion specific example which concerns on this invention. It is a graph which shows the Fourier-transform of the frequency spectrum of an object acoustic signal concerning the 3rd expansion specific example which concerns on this invention. It is a block diagram of the part which concerns on 2nd Embodiment of this invention and is related to the production | generation of an expansion | extension acoustic signal.

  Hereinafter, embodiments of the present invention will be specifically described with reference to the drawings. In each of the drawings to be referred to, the same part is denoted by the same reference numeral, and redundant description regarding the same part is omitted in principle.

<< First Embodiment >>
A first embodiment of the present invention will be described. FIG. 1 is a block diagram showing the overall configuration of the imaging apparatus 1 according to the first embodiment of the present invention. The imaging device 1 includes each part referred to by reference numerals 11 to 18. The imaging device 1 is a digital video camera that can capture still images and moving images. Note that it is possible to interpret that the display unit 16 and / or the speaker 17 is provided in a playback device different from the imaging device 1.

  The imaging unit 11 captures a subject using an imaging element, and acquires a video signal of an image of the subject in cooperation with the video signal processing unit 12. Specifically, the imaging unit 11 includes an imaging device including an optical system (not shown), a diaphragm, and a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. This image sensor photoelectrically converts an optical image representing a subject incident through an optical system and a diaphragm, and outputs an analog electric signal obtained by the photoelectric conversion. An AFE (Analog Front End) (not shown) amplifies an analog signal output from the image sensor and converts it into a digital signal.

  The obtained digital signal is sent to the video signal processing unit 12, and the video signal processing unit 12 generates a video signal of the image of the subject from the digital signal. A video signal expressed in a digital signal format is also called image data. In this specification, image data may be simply referred to as an image. The video signal processing unit 12 can perform various image processing (such as demosaicing processing, edge enhancement processing, noise reduction processing, and image compression processing) on the image data of the subject image.

  The microphone unit 13 includes one or a plurality of microphones, collects sound from a sound source located around the imaging device 1 and converts it into an electrical signal. The obtained electrical signal is sent to the acoustic signal processing unit 14 as an acoustic signal. The acoustic signal processing unit 14 can perform various acoustic signal processing on the acoustic signal, details of which will be described later.

  The recording medium 15 is a nonvolatile memory composed of a semiconductor memory, a magnetic disk, or the like, and can record the video signal generated by the video signal processing unit 12 and the audio signal generated by the audio signal processing unit 14. . The display unit 16 includes a liquid crystal display or the like, and displays an image obtained by photographing with the imaging unit 11, an image recorded on the recording medium 15, and the like. The speaker 17 reproduces and outputs the sound signal generated by the sound signal processing unit 14 and the sound signal recorded on the recording medium 15 as sound.

  The operation unit 18 is a part for the user to perform various operations on the imaging apparatus 1. As shown in FIG. 2, the operation unit 18 includes a shutter button 18a for instructing to capture a still image and a recording button 18b for instructing start and end of moving image capturing. The main control unit 19 comprehensively controls the operation of each part in the imaging apparatus 1 while following the operation content performed on the operation unit 18.

  The number of microphones forming the microphone unit 13 may be 1 or 3 or more. In this embodiment, as shown in FIG. 3, the microphone unit 13 includes two microphones, that is, the microphone 13L. And 13R are assumed. FIG. 4 is an external perspective view of the imaging device 1 provided with the microphones 13L and 13R.

  The microphones 13L and 13R are arranged at different positions on the housing of the imaging device 1. The microphone 13L is arranged on the left side and the microphone 13R is arranged on the right side as viewed from the photographer facing the subject of the imaging apparatus 1. Also, as shown in FIG. 4, the direction from the imaging device 1 toward the subject within the imaging range of the imaging unit 11 is defined as the front, and the opposite direction is defined as the rear. The microphones 13L and 13R are omnidirectional microphones having no directivity. However, microphones having directivity can be employed as the microphones 13L and 13R.

  The microphone 13L converts the sound collected by itself into an electrical signal and outputs a detection signal representing the sound. The microphone 13R converts the sound collected by itself into an electrical signal and outputs a detection signal representing the sound. These detection signals are analog acoustic signals. The analog acoustic signals that are detection signals of the microphones 13L and 13R are converted into digital acoustic signals by A / D converters (not shown), respectively.

  Consider the microphone 13L corresponding to the left channel and the microphone 13R corresponding to the right channel. When distinguishing between the acoustic signal based on the detection signal of the microphone 13L and the acoustic signal based on the detection signal of the microphone 13R, the former is particularly called a left channel acoustic signal and the latter is especially called a right channel acoustic signal. A digital acoustic signal obtained by digitally converting the detection signals of the microphones 13L and / or 13R is referred to as an original acoustic signal. An acoustic signal obtained by subjecting a digital acoustic signal obtained by digitally converting the detection signal of the microphone 13L and / or 13R to a predetermined signal processing (such as signal level adjustment processing by auto level control) is regarded as an original acoustic signal. May be. The original acoustic signal is assumed to be a signal on the time axis. Unless otherwise specified, any acoustic signal in the present embodiment and other embodiments described later can be interpreted as an acoustic signal on the time axis (an acoustic signal expressed in the time domain).

  By the way, in the imaging device 1, the frame rate at the time of moving image shooting is variable, and the frame rate at the time of moving image playback is also variable. The user can set the shooting mode to the normal shooting mode or the high-speed shooting mode via the operation unit 18. Hereinafter, the frame rate at the time of moving image shooting is also referred to as a shooting rate, and the frame rate at the time of moving image playback is also referred to as a playback rate.

  In the normal shooting mode, a moving image is shot at 60 fps (frame per second) as shown in FIG. A moving image shot at 60 fps can be reproduced at the same frame rate (ie, 60 fps). In this case, the captured moving image is displayed on the display unit 16 at a normal reproduction speed. That is, 60 frames taken over 1 second are displayed on the display unit 16 over 1 second.

  In the high-speed shooting mode, a moving image is shot at 600 fps as shown in FIG. A moving image shot at 600 fps can be reproduced at 60 fps. In this case, 600 frames taken over 1 second are displayed on the display unit 16 over 10 seconds. Thereby, substantial slow reproduction can be realized. The specific values of the shooting rate and the playback rate are, of course, exemplary, the shooting rate in the normal shooting mode may be other than 60 fps (for example, 30 fps), and the shooting rate in the high-speed shooting mode may be other than 600 fps (for example, 300 fps). The specific numerical value of the reproduction rate can be changed as the shooting rate is changed.

  In the following description, it is assumed that the target moving image is shot at 600 fps in the high-speed shooting mode, and the target moving image is played back at 60 fps during playback. When the target moving image is shot over α seconds, the original sound signal for α seconds is picked up in the shooting section, and the target moving image shot over α seconds is captured for (10 × α) seconds. When the original sound signal is simply played back slowly when the sound is played back slowly, the sound is reproduced as if the pitch of the sound signal is fluctuated (α is an arbitrary positive number). The pitch is the fundamental frequency of the acoustic signal, and when the sound source is a human voice, the pitch is the fundamental frequency of the acoustic signal due to human vocal cord vibration.

  A method of expanding an acoustic signal while maintaining the pitch (in other words, a method in which the pitch is not changed before and after the expansion process) is known, but such an expansion method is not always appropriate. In the stretching method that maintains the pitch, the acoustic signal is basically stretched by dividing the acoustic signal into a plurality of blocks and repeatedly reproducing the same block a plurality of times. For this reason, a stretching method that maintains the pitch is relatively suitable for the acoustic signal of a human voice (because the pitch does not change and each sound is simply stretched). When applied to music formed with a mixture of frequencies, a sound with an uncomfortable feeling is often generated. In addition, when slow-playing a baseball batting scene or the like, it is considered that the echo processing of the sound at the moment of hitting the ball with the bat matches the reproduced video more.

  In consideration of these, the imaging apparatus 1 is provided with a function of generating an acoustic signal suitable for reproduction of the target moving image from the original acoustic signal. FIG. 7 shows a block diagram of a part particularly related to this function. The sound source type analysis unit 31, the audio signal expansion unit 32, and the audio signal encoding unit 33 shown in FIG. 7 can be provided in the audio signal processing unit 14 of FIG. 1, and the video signal analysis unit shown in FIG. 34 can be provided in the video signal processing unit 12 of FIG.

  The target sound signal is input to the sound source type analysis unit 31 (hereinafter may be abbreviated as the analysis unit 31) and the acoustic signal expansion unit 32 (hereinafter may be abbreviated as the expansion unit 32). The target sound signal is an original sound signal picked up by the microphone unit 13 at the time of shooting the target moving image.

  The analysis unit 31 analyzes the type of the sound source of the signal component included in the target acoustic signal based on the target acoustic signal. In other words, the type of sound signal from the sound source is analyzed based on the target sound signal. For example, whether the sound source of the signal component included in the target acoustic signal is a human voice (in other words, a human vocal cord), music, or an impulse sound (hereinafter referred to as an impulse sound) Or whether it is an animal call. Information representing the analysis result of the analysis unit 31 is sent to the decompression unit 32 as sound source type information.

  On the other hand, the video signal analysis unit 34 analyzes an object included in the target moving image based on the target video signal that is a video signal of the target moving image. For example, it is possible to analyze whether or not a human face exists on the target moving image using face detection processing. Further, for example, it is possible to analyze whether or not the target moving image is an image of a sports scene from the magnitude of the speed of movement of the object on the target moving image. Information representing the analysis result of the video signal analysis unit 34 is sent to the decompression unit 32 as video analysis information.

  The extension unit 32 generates an extended sound signal by extending the target sound signal in time according to the frame rate information. The shooting rate of the target moving image and the playback rate of the target moving image are defined by the frame rate information. In this example, as described above, since the shooting rate of the target moving image is 600 fps and the playback rate of the target moving image is 60 fps, the sound for (10 × α) seconds is obtained from the target sound signal for α seconds. The signal is generated as a stretched acoustic signal.

  A method for generating the extended sound signal from the target sound signal is determined mainly according to the sound source type information, and the method can be determined depending on the video analysis information and the scene setting information. The scene setting information is information indicating the set shooting scene, and the user can set the shooting scene to a desired one using the operation unit 18. For example, when shooting a sports landscape, the user can set the shooting scene to “sports”, and when shooting a subject close to the imaging apparatus 1, the user can set the shooting scene to “macro”. . When the shooting scene is set to “sport”, the imaging apparatus 1 performs shooting of the target moving image under shooting conditions suitable for shooting a sports landscape, and when the shooting scene is set to “macro”. The imaging apparatus 1 performs shooting of the target moving image under shooting conditions suitable for shooting a close subject.

  A method for generating an extended sound signal in accordance with the sound source type information, the video analysis information, and the scene setting information will be described in detail later. Note that the generation of the extended acoustic signal based on the target acoustic signal can be performed for each channel. That is, the expansion unit 32 generates a left channel extended sound signal by extending the left channel target sound signal in time, and expands the right channel target sound signal in time by extending the right channel target sound signal. A signal can be generated. In the following description, the channel is not described separately unless particularly required.

  The acoustic signal encoding unit 33 generates an encoded acoustic signal by encoding the expanded acoustic signal generated by the expansion unit 32 using a predetermined encoding method (for example, AAC (Advanced Audio Coding)). On the other hand, in the video signal processing unit 12 in FIG. 1, the video signal of the target moving image is encoded to generate an encoded video signal. The encoded acoustic signal is recorded on the recording medium 15 together with the encoded video signal of the target moving image while being temporally associated with the encoded video signal of the target moving image.

  At the time of reproduction, the encoded video signal and the encoded audio signal of the target moving image are read from the recording medium 15, and are decoded by the video signal processing unit 12 and the audio signal processing unit 14 to obtain the video signal of the target moving image and A stretched acoustic signal is generated. By sending the video signal obtained by decoding to the display unit 16 at 60 fps, the target moving image is reproduced and displayed at 60 fps, and the decompressed acoustic signal is sent to the speaker 17 to synchronize with the reproduced video of the target moving image. The extended acoustic signal is reproduced as sound.

  FIG. 8 shows a temporal relationship between the target moving image and the extended sound signal during reproduction. The target moving image shot at 600 fps over α seconds is reproduced at 60 fps over (10 × α) seconds during reproduction. On the other hand, the extended acoustic signal for (10 × α) seconds generated from the original sound signal for α seconds collected during the shooting of the target moving image is synchronized with the reproduction of the target moving image at 60 fps. It is reproduced by the speaker 17 over 10 × α) seconds.

[Sound source type analysis method]
A method of analyzing the type of sound source by the analysis unit 31 will be described. A method for determining whether or not an acoustic signal from a specific type of sound source is included in the target acoustic signal in the specific section will be described by paying attention to a specific section included in all sections in which the target acoustic signal exists. Note that the analysis unit 31 determines whether the left channel and the right channel in the specific section are based on the left channel target acoustic signal alone or only on the right channel target acoustic signal. It can be determined whether or not an acoustic signal from a specific type of sound source is included in the target acoustic signals of the channel and the right channel. Alternatively, the determination can be made based on the target acoustic signals of the left channel and the right channel in the specific section.

  Whether or not a target voice signal in a specific section includes a sound signal by human voice is disclosed in a known utterance section detection method (for example, Japanese Patent Laid-Open No. 10-257596) used in speech recognition processing or the like. Method). Specifically, for example, it is possible to detect whether or not an acoustic signal based on a human voice is included in a target acoustic signal in a specific section by a method based on pitch extraction using autocorrelation processing. A section including an acoustic signal from a human voice is also called an utterance section.

  Considering a case where a digital acoustic signal for 1024 samples is included in a specific section, a method for detecting a speech section that can be employed by the analysis unit 31 will be described. The signal value of the t-th digital acoustic signal among the digital acoustic signals for 1024 samples forming the target acoustic signal in the specific section is represented by x (t). t takes an integer value between 1 and 1024.

  As shown in FIG. 9, the analysis unit 31 calculates the autocorrelation using a block composed of the first to 128th digital sound signals as a reference block. That is, an evaluation block composed of 128 consecutive digital sound signals is defined within a specific section, and the correlation between the reference block and the evaluation block is obtained while sequentially shifting the temporal position of the evaluation block. More specifically, the autocorrelation value S (p) is calculated according to the following formula (1). The autocorrelation value S (p) is a function of a variable p that determines the position of the evaluation block, and p is an integer that satisfies 0 ≦ p ≦ (1024−128).

FIG. 10 shows the variable p dependency of the calculated autocorrelation value S (p). In FIG. 10, the horizontal axis represents the variable p. FIG. 10 corresponds to the case where the target sound signal in the specific section includes a sound signal based on a human voice. When the target acoustic signal includes a pitch due to human vocal cord vibration, the autocorrelation value S (p) takes a large value periodically. When the autocorrelation value S (p) periodically exceeds a predetermined threshold TH _A and the fundamental frequency that is the reciprocal of the period falls within the predetermined frequency range R _VOICE , the analysis unit 31 It can be determined that the sound signal includes a sound signal from a human voice (that is, it can be determined that the specific section is a speech section). Otherwise, the target sound signal in the specific section is included in the target sound signal. It can be determined that an audio signal from a human voice is not included. For example, when the interval of the variable p satisfying the inequality “S (p)> TH _A ” is constant (or substantially constant), the autocorrelation value S (p) periodically exceeds a predetermined threshold value TH _A. to decide. Since the pitch (fundamental frequency) due to human vocal cord vibration is approximately in the band of 80 to 270 Hz, the lower limit frequency and the upper limit frequency of the frequency range R _VOICE are set to 50 Hz and 300 Hz, respectively.

Whether or not the target acoustic signal in the specific section includes an acoustic signal based on music can be detected by the same method as the above-described detection method of the speech section. This is because the sound signal by music also has a certain periodicity. However, in general, the fundamental frequency of an acoustic signal due to music is higher than that of an acoustic signal due to human vocal cord vibration. Therefore, the analysis unit 31 has the autocorrelation value S (p) periodically exceeding the predetermined threshold TH _A and the fundamental frequency that is the reciprocal of the period exceeds the upper limit frequency of the predetermined frequency range R _VOICE . In this case, it can be determined that the target acoustic signal in the specific section includes an acoustic signal based on music.

In addition, even if the fundamental frequency of the acoustic signal of music is about the same as that of a human voice, it is determined whether or not a spectrum envelope (envelope) peculiar to the human voice can be seen in the target acoustic signal. Thus, it can be distinguished whether the target sound signal is a sound signal of a human voice or a sound signal of music. Due to the influence of resonance, the frequency spectrum of an acoustic signal generated by a human voice tends to have a peak at a specific frequency. On the other hand, such a tendency is not seen in the acoustic signal of music. Therefore, it is determined that the autocorrelation value S (p) for the target acoustic signal in the specific section periodically exceeds the predetermined threshold TH _A , and the target acoustic signal includes an acoustic signal based on human voice or music. By distinguishing whether the above-mentioned tendency exists in the target acoustic signal, it is possible to distinguish whether the target acoustic signal is an acoustic signal based on a human voice or an acoustic signal based on music. good.

Further, the analysis unit 31 may determine whether the target acoustic signal includes an acoustic signal due to the impulse sound based on the magnitude of the signal value or the power change amount in the target acoustic signal on the time axis. it can. Specifically, for example, when a section where the amount of change per unit time of the signal value or power in the target acoustic signal exceeds a predetermined threshold TH _B exists in a specific section, an impulse sound is included in the section. Can be determined, and it can be determined that the target acoustic signal in the specific section includes the acoustic signal due to the impulse sound. As an impulse sound, a hitting sound at the moment of hitting a ball with a baseball bat, a sound of tapping a drum, and the like are assumed.

  Moreover, the analysis part 31 can also determine whether the acoustic signal by the call of an animal is contained in the target acoustic signal in a specific area based on the target acoustic signal in a specific area. The above determination can be made by detecting a section where an animal cry exists based on the characteristics of the animal's cry as in the case of detecting the utterance section based on the characteristics of the human voice.

  Specifically, the animal cry is a dog or cat cry. In the case of dog calls, a database on dog calls is created in advance by learning various dog calls, and the target acoustic signal in a specific section is checked by comparing the target acoustic signal in the specific section with the database. It is possible to determine whether or not a sound signal from a dog cry is included. This determination may be performed in consideration of the target video signal. That is, for example, based on the target video signal in the specific section, the video signal analysis unit 34 analyzes whether the target moving image in the specific section includes the dog image, and the analysis result is also taken into account. It may be determined whether or not the target sound signal in FIG.

[Method for generating extended acoustic signal]
Next, a method for generating a stretched acoustic signal by the stretching unit 32 will be described. The decompressing unit 32 generates a decompressed acoustic signal by subjecting the target acoustic signal to decompression processing adapted to sound source type information and the like. The expansion process related to an acoustic signal refers to a process of increasing the signal length of the acoustic signal by stretching the acoustic signal to be expanded in the time direction. The signal length of the acoustic signal refers to the time length of the section where the acoustic signal exists. It is assumed that the time length of the specific section before the decompression process is β seconds (β is an arbitrary positive number). In this example, since the reproduction rate is 1/10 of the shooting rate, the time length of the specific section after the expansion process is (10 × β) seconds, and the signal length of the target acoustic signal for β seconds in the specific section is Then, it is stretched by a factor of 10 by the stretching process to generate a stretched acoustic signal having a signal length of (10 × β) seconds. Of course, the expansion time (the time extended by the expansion process) is changed in accordance with the reproduction rate, and is increased as the reproduction rate becomes slower, for example.

  However, if the sound signal is stretched by an amount corresponding to the playback rate accurately, a strange sound may be played back. Therefore, the time corresponding to the difference between the shooting rate and the playback rate is matched with the extension time. There is no necessity. That is, for example, when the playback rate is 1/10 of the shooting rate, as shown in FIG. 11, the target acoustic signal for β seconds is stretched 6 times on the time axis to obtain an acoustic signal for (6 × β) seconds. And (10 × β) seconds of extended sound signals are generated by connecting (4 × β) seconds of silence signals to (6 × β) seconds of sound signals. good. A silence signal refers to an acoustic signal having a signal level and power of zero (or substantially zero).

  As expansion processing that can be employed by the expansion unit 32, simple expansion processing, pitch maintenance expansion processing, echo processing, and repeat processing will be described below.

  Although a specific example will be described later, when the target sound signal includes sound signals from a plurality of different sound sources (for example, the target sound signal includes a sound signal based on human voice and a sound signal based on music) 12), as shown in FIG. 12, the decompressing unit 32 extracts the acoustic signals from the plurality of sound sources as a plurality of separated acoustic signals individually from the target acoustic signal, and sets the type of the sound source of each separated acoustic signal. After the analysis, each of the separated acoustic signals is subjected to a decompression process corresponding to the type of the sound source of each separated acoustic signal, and then a plurality of separated acoustic signals are synthesized to generate a decompressed acoustic signal.

  Accordingly, simple extension processing, pitch maintenance extension processing, and the like are executed individually for each separated acoustic signal. Therefore, assuming that simple extension processing, pitch maintenance extension processing, and the like are performed on the separated acoustic signal, the extension processing will be described. When the target acoustic signal includes only the acoustic signal from a single sound source, the separated acoustic signal based on the target acoustic signal is the target acoustic signal itself. FIG. 12 is an image diagram of the generation process of the extended acoustic signal in the case where the target acoustic signal includes acoustic signals from two types of sound sources (it is only an image diagram and the validity of the waveform and the like in FIG. 12) Note that is low).

-Simple extension processing-
The simple decompression process will be described. A separated acoustic signal in a specific section to be subjected to simple extension processing is called an acoustic signal A _1, and an acoustic signal obtained by performing simple extension processing on the acoustic signal A ₁ is called an acoustic signal B ₁ . In this example, the length of the section in which the acoustic signal B ₁ exists is 10 times that of the acoustic signal A ₁ . FIG. 13 is an image diagram of simple decompression processing. On the time axis, the acoustic signal B ₁ is obtained by simply stretching the acoustic signal A ₁ 10 times. Therefore, the signal component of the frequency f included in the acoustic signal A ₁ is converted into the signal component of the frequency (f / 10) in the acoustic signal B ₁ . When the simple extension process is performed, the pitch is naturally changed and the pitch is changed.

As described with reference to FIG. 11, (4 × β) seconds of silence for the (6 × β) seconds of the acoustic signal obtained by simply extending the acoustic signal A ₁ by 6 times. By connecting the signals, the acoustic signal B ₁ for (10 × β) seconds may be generated.

--Pitch maintenance and extension process--
The pitch maintaining / extending process will be described. A separated acoustic signal in a specific section to be subjected to the pitch maintaining / extending process is referred to as an acoustic signal A _2, and an acoustic signal obtained by performing the pitch maintaining / extending process on the acoustic signal A ₂ is referred to as an acoustic signal B ₂ . In this example, the length of the section in which the acoustic signal B ₂ exists is 10 times that of the acoustic signal A ₂ .

In the pitch maintaining / extending process, the acoustic signal is extended so that the pitch does not change between the acoustic signals A ₂ and B ₂ . As the expansion method, a known speech speed conversion method can be used. FIG. 14 is an image diagram of the pitch maintaining / extending process. For example, the specific section is divided into first to Nth blocks with a block length corresponding to the pitch of the acoustic signal A ₂ (N is an integer of 2 or more), and the acoustic signal A ₂ in the first block is and 10 times repeated signal, and the signal was repeated acoustic signal a ₂ 10 times in the second block, ..., a (N-1) th signal repeated acoustic signal a ₂ 10 times in a block of, the The acoustic signal B ₂ can be generated by connecting the signals obtained by repeating the acoustic signal A ₂ in the N blocks 10 times in this order.

As described with reference to FIG. 11, the signal obtained by repeating the acoustic signal A ₂ in the first block six times, the signal obtained by repeating the acoustic signal A ₂ in the second block six times,... , A signal obtained by repeating the acoustic signal A ₂ in the (N−1) th block 6 times, a signal obtained by repeating the acoustic signal A ₂ in the Nth block 6 times, and a silence signal for (4 × β) seconds and it may generate an acoustic signal B ₂ by connecting in this order. However, in this method, silent signal is biased in the second half of the audio signal B _2. In order to avoid such a bias, a signal obtained by repeating the acoustic signal A ₂ in the first block six times, a silence signal for (4 × B _{L [1]} ) seconds, and an acoustic signal in the second block A signal obtained by repeating A ₂ 6 times, a silence signal corresponding to (4 × B _{L [2]} ) seconds, a signal obtained by repeating the acoustic signal A ₂ in the (N−1) th block 6 times, , (4 × B _{L [N-1]} ) seconds of silence signal, a signal obtained by repeating the acoustic signal A ₂ in the Nth block 6 times, and (4 × B _{L [N]} ) seconds of silence signal May be generated in this order to generate the acoustic signal B ₂ . Here, B _{L [i]} represents the block length in the i-th block (that is, the time length of the i-th block) (i is an integer).

--Echo processing--
The echo process will be described. A separated acoustic signal in a specific section to be subjected to echo processing is called an acoustic signal A _3, and an acoustic signal obtained by performing echo processing on the acoustic signal A ₃ is called an acoustic signal B ₃ . In this example, the section length in which the acoustic signal B ₃ exists is 10 times that of the acoustic signal A ₃ .

In the echo processing, the same acoustic signal as the acoustic signal A ₃ is repeated a plurality of times while gradually reducing the signal level. FIG. 15 is an image diagram of echo processing. Acoustic signal B ₃ is an echo signal _{A 3 [1], A 3} [2], A 3 [3], A 3 [4], A 3 [5], A 3 [6], A 3 [7], A _{3 [8]} , A _{3 [9]} and A _{3 [10]} are connected in this order. Here, the signal waveform of the echo signal A _{3 [i] and} the signal waveform of the acoustic signal A ₃ are similar, and the signal level and power of the echo signal A _{3 [i + 1]} are the same as those of the echo signal A _{3 [i]} . It is smaller than the signal level and power (i is an integer). Therefore, when the acoustic signal B ₃ is reproduced, the acoustic signal A ₃ is repeatedly reproduced while the volume gradually decreases. For example, if the acoustic signal A ₃ is a sound of “Kakkin” that is a batting sound, the sound of “Kakkin” is repeatedly reproduced 10 times while the volume is gradually reduced by the reproduction through the echo process. become.

As described with reference to FIG. 11, the echo signals A _{3 [1]} , A _{3 [2]} , A _{3 [3]} , A _{3 [4]} , A _{3 [5]} and A _{3 [6]} May be generated as the acoustic signal B ₃ by connecting a signal connecting the two and a silence signal for (4 × β) seconds. Depending on the playback rate, the number of echoes (that is, the number of times the echo signal A _{3 [i]} is repeated), the time for applying the echo (that is, the time that the echo signal A _{3 [i]} is repeated), and / or The attenuation rate of the echo signal (that is, the attenuation rate of the signal level of the echo signal A _{3 [i + 1]} with respect to the signal level of the echo signal A _{3 [i]} ) may be changed.

-Repeat processing-
The repeat process will be described. A separated acoustic signal in a specific section to be subjected to the repeat processing is referred to as an acoustic signal A _4, and an acoustic signal obtained by performing the repeat processing on the acoustic signal A ₄ is referred to as an acoustic signal B ₄ . In this example, the length of the section in which the acoustic signal B ₄ exists is 10 times that of the acoustic signal A ₄ .

In the repeat processing, the same acoustic signal as the acoustic signal A ₄ is simply repeated a plurality of times. That is, the acoustic signal B ₄ is a repeat signal A _{4 [1]} , A _{4 [2]} , A _{4 [3]} , A _{4 [4]} , A _{4 [5]} , A _{4 [6]} , A _{4 [7 ]} , A4 _[8] , A4 _[9] and A4 _[10] are connected in this order, and each of the repeat signals A4 _{[1] to} A4 _[10] includes the signal level. This is the same as the acoustic signal A ₄ . Therefore, for example, when the acoustic signal A ₄ is an acoustic signal of a certain music, at the time of reproducing the acoustic signal B ₄ obtained through the repeat process, the music is not accompanied by a change in pitch (10 × β). Playback is repeated at a normal playback speed in a specific section of seconds.

  The decompression unit 32 changes the content of the decompression process to be performed on the separated acoustic signal according to the sound source type information and the like. For example, when it is determined that the focused sound source of the separated acoustic signal is a human voice, a pitch maintaining / extending process is performed on the focused separated acoustic signal, and the focused sound source type of the separated acoustic signal Is determined to be an impulse sound, echo processing can be performed on the focused separated acoustic signal.

  For example, when it is determined that the type of the sound source of the separated sound signal of interest is music, repeat processing can be performed on the separated sound signal of interest, or the separated sound signal of interest is It may be deleted (that is, the signal component of music may be excluded from the extended acoustic signal), or the signal level of the separated separated acoustic signal may be reduced. To delete a specific acoustic signal, the signal component of the specific acoustic signal is deleted from the target acoustic signal during the expansion process so that the signal component of the specific acoustic signal is not included in the expanded acoustic signal. Refers to an operation. In this way, the expansion processing method can be changed depending on whether the type of the sound source of the separated acoustic signal is a human voice. Further, the contents of the decompression process may be determined in accordance with the video analysis information (a use example of the video analysis information will be described in detail in a second specific decompression example described later).

  Next, first to fourth expansion specific examples will be described as specific examples according to various situations of the expansion processing based on the sound source type information and the like.

[First example of expansion]
A first example of expansion will be described. In the first extension specific example, it is assumed that a batter hitting a ball with a bat in a baseball game was shot as a target moving image. The target acoustic signal includes an acoustic signal of a cheer of a person who is cheering a baseball player in addition to an acoustic signal of a hitting sound generated when the ball is hit with a bat.

  The analysis unit 31 and the decompression unit 32 analyze the target acoustic signal to separately extract the acoustic signal of the hitting sound and the cheering acoustic signal from the target acoustic signal as separated acoustic signals, and perform the extraction on the separated acoustic signal of the impacting sound. , While performing echo processing, performs pitch maintenance expansion processing on the separated sound signal of cheers. Then, the extended acoustic signal is generated by synthesizing the separated acoustic signal of the hitting sound after the echo processing and the separated acoustic signal of the cheer after the pitch maintaining / extending processing.

  FIG. 16A is an image diagram of normal reproduction of the target sound signal and target moving image under the assumption of the first extension specific example, and FIG. 16B is an extension sound according to the first extension specific example. It is an image figure of slow reproduction of the object moving picture accompanied with reproduction of a signal. During slow playback of the target video, the cheering sound is played slowly while maintaining the pitch, while in the surrounding section where the moment of striking is displayed, the sound of “Kakein”, which is a striking sound, is accompanied by a gradual decrease in volume. Output repeatedly. In this scene, since the moment of hitting is the most important timing, it is desirable to reduce the volume of cheers as much as possible in the section including the moment of hitting.

A method for generating the separated acoustic signal and the extended acoustic signal in the first extension specific example will be described more specifically. As shown in FIG. 17, all the sections of the target acoustic signal are classified into three sections P _1A , P _1B and P _1C , and only the cheering sound signal exists in the sections P _1A and P _1C , and in the section P _1B Assume that there is a sound signal of hitting sound and cheers.

Since the target acoustic signals in the sections P _1A and P _1C include only a cheering (ie, human voice) acoustic signal, the analysis unit 31 performs the target acoustics in the sections P _1A and P _1C by the method described above. It can be easily known that the signal includes an audio signal by human voice. Furthermore, the analysis unit 31 considers the section P _1B as a specific section, and uses the above-described method for determining whether or not the target acoustic signal in the specific section includes an acoustic signal due to an impulse sound, thereby It can be known that the target acoustic signal in P _1B includes the acoustic signal due to the impulse sound.

Since the target acoustic signals in the sections P _1A and P _1C include the human voice acoustic signal, the analysis unit 31 or the expansion unit 32 includes the human voice acoustic signal in the target acoustic signal in the section P _1B . Can be guessed. Decompression unit 32, in order to separate and extract the acoustic signals of human voice sound signal and an impulse sound from the target sound signal in the interval P _1B (impact sound in this example), the target sound signal on the time axis in the section P _1B Is subjected to Fourier transform to generate a target acoustic signal on the frequency axis in the section P _1B , that is, a frequency spectrum of the target acoustic signal in the section P _1B . A discrete Fourier transform is used as the Fourier transform.

In the graph in FIG. 18A, each spectrum component of the frequency spectrum 310 of the target acoustic signal in the section P _1B is shown. The frequency spectrum 310 is obtained by adding the spectrum component of the human voice represented by the solid line 311 and the spectrum component of the impulse sound represented by the broken line 312. The spectral component 311 of the human voice periodically varies with frequency changes, while the spectral component 312 of the impulse sound corresponding to the addition of a wide range of frequency components periodically varies with frequency changes. It does not have a natural property.

Paying attention to such a property, the decompression unit 32 performs a Fourier transform on the frequency spectrum 310 once again. It is assumed that an acoustic signal is converted into an acoustic signal on the F axis by performing Fourier transform on the acoustic signal on the frequency axis. The graph in FIG. 18B represents the target acoustic signal 320 on the F axis in the section P _1B . The target acoustic signal 320 on the F axis is obtained by adding the signal component of the human voice represented by the solid line 321 and the signal component of the impulse sound represented by the broken line 322. Due to the above-described properties, the human voice signal component and the impulse sound signal component exist separately on the F-axis. When a certain target acoustic signal on the frequency axis fluctuates periodically with respect to a change in frequency, the target acoustic signal on the F axis shifts to a higher frequency side when the period of the variation becomes shorter. And

  The decompression unit 32 can determine that the signal component 321 is a signal component of a human voice if the frequency on the F axis of the signal component 321 is within a predetermined audio frequency range, otherwise, It can be determined that the signal component 321 is not a human voice signal component. Now, it is assumed that the frequency of the signal component 321 on the F axis is within a predetermined audio frequency range.

In the expansion unit 32, the signal component (that is, the signal component 321) located on the high frequency side of the F axis in the target acoustic signal 320 on the F axis is a human voice signal component, and the F axis The signal component (that is, the signal component 322) positioned on the low frequency side is regarded as the signal component of the impulse sound, and the former signal component (that is, the signal component 321) and the latter signal component (that is, the signal component 322). ) Twice separately. As the inverse Fourier transform, a discrete inverse Fourier transform is used. As a result, a separated acoustic signal on the time axis based on the human voice in the section P _1B is generated from the signal component 321, and a separated acoustic signal on the time axis based on the impulse sound in the section P _1B is generated from the signal component 322. . In addition, the separated acoustic signal on the time axis based on the human voice in the section P _1A (ie, the target acoustic signal in the section P _1A ) and / or the separated acoustic signal on the time axis based on the human voice in the section P _1C (ie, the section) from the target sound signal) at P _1C, it may be estimated separation acoustic signal on the time axis by the human voice in the section P _1B.

Different extension processes are performed on the separated voice signals of the human voice and the impulse sound in the section P _1B obtained through the inverse Fourier transform. On the other hand, since the target acoustic signals in the sections P _1A and P _1C include only the sound signal of the human voice, the target acoustic signals themselves are subjected to pitch maintenance / extension processing for the sections P _1A and P _1C . . That is, the decompression unit 32 performs a pitch maintenance decompression process on the target acoustic signal in the section P _1A, the separated voice signal of the human voice in the section P _1B, and the target acoustic signal in the section P _1C and connects them in time. The first component of the acoustic signal is generated, while the second component of the extended acoustic signal is generated by performing echo processing on the separated acoustic signal of the impulse sound in the section P _1B . Here, the first component of the extended acoustic signal includes the acoustic signal in the entire section, but the second component of the extended acoustic signal includes only the acoustic signal in the section P _1B . The extension unit 32 generates a final extended sound signal by combining the first component and the second component of the extended sound signal.

  By playing the extended acoustic signal obtained as described above together with the slow playback of the video, the baseball batting scene can be played as a powerful scene.

[Second specific example]
A second example of expansion will be described. In the second extension specific example, it is assumed that a child playing in a park or the like is captured as a target moving image. A child to be photographed is particularly called a person of interest. Then, it is assumed that the target acoustic signal includes an acoustic signal of a voice of another person in the park (hereinafter referred to as a non-attention person) in addition to the acoustic signal of the voice of the person of interest.

  FIG. 19A is an image diagram of normal reproduction of the target sound signal and target moving image under the assumption of the second extension specific example, and FIG. 19B is an extension sound according to the second extension specific example. It is an image figure of slow reproduction of the object moving picture accompanied with reproduction of a signal. At the time of slow playback of the target moving image, the voice of the person of interest is played back slowly while maintaining the pitch.

  In the second extension specific example, the analysis result of the target video signal is used in addition to the analysis result of the target sound signal in generating the extension sound signal. Specifically, the processing is as follows.

  Based on the target video signal, the video signal analysis unit 34 determines whether or not a human face having a size equal to or larger than the reference face size is included in the target moving image. Now, it is assumed that the face of the target person exists on the target moving image, and the size of the face of the target person on the target moving image is equal to or larger than a predetermined reference face size. Then, the video signal analysis unit 34 determines that a face (human face) having a size equal to or larger than the reference face size is included in the target moving image, and expands the video analysis information including the determination result. Send to. When such video analysis information is sent, the expansion unit 32 performs not only pitch maintenance expansion processing but also front sound enhancement processing on the target audio signal based on the video analysis information and the sound source type information from the analysis unit 31. And the target acoustic signal after the processing is output as an extended acoustic signal. When the target moving image does not include a human face having a size larger than the reference face size, the front sound enhancement process is not performed on the target sound signal.

  The front sound enhancement process is a process of enhancing a signal component of a sound that has arrived from the front direction of the imaging device 1 (hereinafter referred to as a front sound) in the target sound signal, or a sound that has arrived from another direction (hereinafter, referred to as a front sound). This is processing for reducing signal components of non-frontal sound. Alternatively, both the former process and the latter process may be executed in the front sound enhancement process.

  For example, as shown in FIG. 20, the midpoint between the diaphragm center of the left channel microphone 13L and the diaphragm center of the right channel microphone 13R is the origin O, the straight line connecting both diaphragm centers is the X axis, and the X axis A straight line that is orthogonal to and passes through the origin O is defined as the Y axis. The XY coordinate plane is a coordinate plane having the X axis and the Y axis as coordinate axes. Furthermore, the direction from the microphone 13L toward the microphone 13R is defined as the positive direction of the X axis, and the direction from the origin O toward the positive side of the Y axis is defined as the front side for the imaging apparatus 1 (see also FIG. 4). . In FIG. 20, line segments 331 and 332 are line segments that pass through the origin O and intersect with the Y axis at an angle of 30 °. However, the line segment 331 extends from the origin O toward the first quadrant on the XY coordinate plane, and the line segment 332 extends from the origin O toward the second quadrant on the XY coordinate plane. The Y axis is substantially parallel to the optical axis of the imaging unit 11, and an object located within a range of 60 ° that crosses when moving from the line segment 331 to the line segment 332 is generally the imaging target of the imaging unit 11. For simplification of explanation, the existence of the Z-axis direction orthogonal to the X-axis and the Y-axis is ignored, but actually, the imaging range of the imaging unit 11 extends in the Z-axis direction.

  Sound coming from a sound source located in the first quadrant of the XY coordinate plane and closer to the Y axis than the line segment 331 and sound coming from the sound source located in the second quadrant of the XY coordinate plane and closer to the Y axis than the line segment 332 Sounds coming from the sound sources located are considered as front sounds, and sounds from other sound sources are considered as non-front sounds. In the front sound enhancement process, based on the phase difference between the target acoustic signal of the left channel and the target acoustic signal of the right channel, the acoustic signal component of the front sound is enhanced among the target acoustic signals of the left channel and the right channel, and / or The acoustic signal component of the non-frontal sound is reduced (the acoustic signal component of the non-frontal sound may be completely deleted). As a method for enhancing or reducing the signal component of the sound coming from a specific direction based on the phase difference information, any method including a known method can be used.

  When the extended acoustic signal obtained through the pitch maintaining / extending process and the front sound emphasizing process as described above is reproduced, the volume of the voice of the person of interest is that of the non-person of interest while the pitch of the voice of the person of interest is maintained. On the other hand, it becomes louder and it becomes easier to hear the voice of the person of interest.

  Note that the above-described front sound enhancement processing may be performed only when a registered person's face is detected from the target moving image. That is, a face image of a registered person to be a person of interest is registered in the imaging device 1 in advance, and the video signal analysis unit 34 compares the face image with images of each part of the target moving image based on the target video signal. By doing so, it is detected whether or not the face of the registered person exists on the target moving image. The front sound enhancement process described above may be performed only when it is determined that the face of the registered person exists on the target moving image.

[Third specific example of expansion]
A third decompression example will be described. In the third extension specific example, it is assumed that a state in which a target person runs through a goal point during an athletic meet is photographed as a target moving image. The target acoustic signal includes an acoustic signal of a “goal” call (hereinafter referred to as “goal utterance”) by a referee of an athlete race, an acoustic signal from a cheer of a person cheering in the vicinity, and music ( Hereinafter, it is assumed that an acoustic signal of BGM) is included. In the target sound signal, the signal level of the sound signal of the goal utterance is sufficiently higher than that of the cheer.

  The analyzing unit 31 and the decompressing unit 32 analyze the target acoustic signal, and separately extract the goal utterance acoustic signal, the cheering acoustic signal, and the BGM acoustic signal from the target acoustic signal as separated acoustic signals, thereby separating the goal utterance. Pitch maintenance expansion processing (or echo processing) is applied to the acoustic signal, pitch maintenance expansion processing is applied to the cheering separation acoustic signal while reducing the volume, and repeat processing is performed on the BGM separation acoustic signal. Apply. Then, an extended acoustic signal is generated by synthesizing the separated acoustic signals after the processing. The repeat processing may be performed after the volume of the BGM separated acoustic signal is reduced, or the BGM separated acoustic signal may be deleted.

  FIG. 21A is an image diagram of normal reproduction of the target sound signal and the target moving image under the assumption of the third extension example, and FIG. 21B is an extension sound according to the third extension example. It is an image figure of slow reproduction of the object moving picture accompanied with reproduction of a signal.

A method for generating the separated acoustic signal and the extended acoustic signal in the third extension specific example will be described more specifically. As shown in FIG. 22, all sections of the target acoustic signal are classified into three sections P _2A , P _2B and P _2C , and only the cheering and BGM acoustic signals exist in the sections P _2A and P _2C , and the section P _2B Suppose that there is an acoustic signal of goal utterance in addition to the cheering and BGM acoustic signals.

First, the expansion method for the section P _2B will be described. The analysis unit 32 considers the section P _2B as a specific section, and uses the above-described method to determine whether the target acoustic signal of the section P _{2B includes} an acoustic signal based on a human voice, and the section It is possible to detect whether or not the P _2B target acoustic signal includes a musical acoustic signal. Under the assumption in the third extension specific example, it is detected that the target sound signal in the section P _2B includes sound signals based on human voice and music.

Decompression unit 32, in order to separate and extract the acoustic signals of the acoustic signals and music human voice from the target sound signal in the interval P _2B (BGM in this example), to subject the acoustic signal on the time axis in the section P _2B By performing Fourier transformation, a target acoustic signal on the frequency axis in the section P _2B , that is, a frequency spectrum of the target acoustic signal in the section P _2B is generated.

The frequency spectrums 361, 362, and 363 shown in the graphs of FIGS. 23A, 23B, and 23C are the frequency spectrum of the acoustic signal due to the goal utterance, the frequency spectrum of the acoustic signal due to the cheer, and the acoustic signal according to BGM, respectively. Is the frequency spectrum. Actually, the sum of the spectra 361 to 363 is generated as the frequency spectrum of the target acoustic signal in the section P _2B , so the spectra 361 to 363 cannot be separated on the frequency axis.

However, in the target acoustic signal, the signal level of the goal utterance is sufficiently higher than that of the cheers, and the fundamental frequency of the human voice is much lower than that of the music. Considering this, the decompression unit 32 performs Fourier transform once again on the frequency spectrum of the target acoustic signal in the section P _2B , which is a combined spectrum of the spectra 361 to 363. The graph in FIG. 24A represents the target acoustic signal 370 on the F axis in the section P _2B . The target acoustic signal 370 on the F-axis is a sum of the human voice signal component represented by the curve 371 and the music signal component represented by the curve 372 (that is, the BGM signal component). Since the fundamental frequency of human voice is much lower than that of music, the signal component of human voice and the signal component of music exist separately on the F axis.

  The signal component of the human voice represented by the curve 371 includes a signal component due to goal utterance with a relatively high signal level and a signal component due to a cheer with a relatively low signal level. A broken line 381 in FIG. 24B represents the former signal component, and broken lines 382 and 383 in FIG. 24C represent the latter signal component. Since the goal utterance is formed by the voice of one person, the cheer is formed by the voices of multiple persons, so the spread of the signal component of the cheer on the F axis is larger than that of the goal utterance. It is getting bigger.

  Regions on the F axis where the signal components within the broken lines 381, 382, and 383 exist are represented by reference numerals 391, 392, and 393, respectively (see FIGS. 24B and 24C). On the F axis, the regions 391 to 393 are regions that do not overlap with each other, the region 393 is located on the higher frequency side than the region 391, and the region 391 is located on the higher frequency side than the region 392.

  When the frequency of the signal component is within a predetermined audio frequency range on the F axis, the decompression unit 32 can determine that the signal component is a signal component of a human voice, otherwise, It can be determined that the signal component is not a signal component of a human voice. Now, it is assumed that the signal component 371 is within the audio frequency range while the signal component 372 is not within the audio frequency range. Further, when the maximum level of the signal component 371 is larger than a predetermined reference level and the spread of the signal component 371 on the F axis is larger than the predetermined reference spread, the signal component 371 includes an acoustic signal based on the main sound and the non-main sound. It can be determined that the acoustic signals by are mixed. Assume that it is determined that such a mixture has occurred. The main audio corresponds to the goal audio, and the non-main audio corresponds to cheers. Of the signal component 371, a portion having a signal level equal to or higher than the reference level is a signal component in the region 391, and a portion having a signal level lower than the reference level is a signal component in the regions 392 and 393. Suppose there is.

In this case, the decompressing unit 32 regards the signal component 372 as a signal component of music (or any signal component other than a human voice), and performs the inverse Fourier transform twice on the signal component 372 to obtain the section P. A separated acoustic signal on the time axis of BGM in _2B is generated. On the other hand, goal utterance in the section P _2B is performed by performing inverse Fourier transform twice on the signal component 371 having a signal level equal to or higher than the reference level (that is, the signal component in the region 391). Of the signal component 371 and the signal component 371 that does not have a signal level equal to or higher than the reference level (that is, the signal components in the regions 392 and 393) twice inverse Fourier By performing the conversion, a separated acoustic signal on the timeline of cheers in the section P _2B is generated. However, since the signal component in the area 391 on the F-axis includes a cheering acoustic signal component, the separated acoustic signal on the time axis of the goal utterance generated here is actually a cheering signal. An acoustic signal component is also included.

On the other hand, since the target acoustic signals in the sections P _2A and P _2C include only the cheering acoustic signal and the BGM acoustic signal, they can be easily separated. That is, by twice the Fourier transform of the target sound signal on the time axis in the section P _2A, converts the target sound signal in the interval P _2A to the signal on the F-axis. Then, on the assumption that the target acoustic signal in the section P _2A includes human voice and music acoustic signals, the signals within the voice frequency range among the target acoustic signals on the F axis in the section P _2A While the component is regarded as a signal component of a human voice (ie cheer), a signal component not within the voice frequency range is regarded as a signal component of music (ie BGM), and a human voice on the F axis Inverse Fourier transform is performed twice for each of the signal component and the music signal component. As a result, in the section P _2A , a separated acoustic signal on the time axis based on the human voice is generated from the signal component of the human voice on the F axis, and separated on the time axis by the BGM from the signal component of the music on the F axis. An acoustic signal is generated. The same applies to the section P _2C .

After generating the respective separation acoustic signal on the time axis in each section, the expansion section 32 repeats the separation acoustic signal BGM in the interval P _2A in isolation acoustic signal cheering in the interval P _2A while performing pitch maintain decompression By performing processing and adding the processed signals, an extended acoustic signal in the section P _2A is generated. However, as described above, the BGM separated acoustic signal in the section P _2A may be reduced or deleted in the course of the expansion process. The same applies to the separated BGM acoustic signals in the sections P _2B and P _2C .
Then, the decompression unit 32 performs a goal utterance and repeating process to separate the acoustic signals of BGM to separate the acoustic signal in the interval P _2B while performing pitch maintaining decompression of cheering in the interval P _2B, sums them after treatment Thus, the extended acoustic signal in the section P _2B is generated. However, as described above, the volume of the cheering separated acoustic signal in the section P _2B may be reduced in the process of the expansion process.
Furthermore, decompression section 32, section by subjecting the repeating process to separate the acoustic signals of BGM in the interval P _2C for the separation acoustic signal cheering in the interval P _2C while performing pitch maintain decompression processing, summing them after treatment generating a decompressed audio signal at P _2C.
Finally, the extension unit 32 completes the extended acoustic signal of all sections by connecting the extended acoustic signal in the section P _2A, the extended acoustic signal in the section P _{2B, and} the extended acoustic signal in the section P _2C in this order.

  By playing back the extended acoustic signal obtained as described above, the goal utterance to be noted is emphasized in a state where the pitch of the goal utterance and the cheer is maintained, and a realistic reproduction is realized. In addition, the BGM is reproduced without a sense of incongruity.

[Fourth specific example]
Further, the content of the expansion processing performed by the expansion unit 32 may be changed according to the scene setting information. For example, when the shooting scene pointed to by the scene setting information is “sports”, by performing expansion processing (for example, pitch maintenance expansion processing) on the sound signal of a human voice that seems to be a cheering voice When the extended sound signal includes a cheer sound signal and the shooting scene indicated by the scene setting information is “macro”, the sound signal of the surrounding sound including the human voice is only composed of the expand sound signal. You may make it exclude.

  Further, the shooting scene determination may be performed from the target video signal without referring to the scene setting information. That is, for example, the optical flow of the target moving image is derived based on the target video signal, and the magnitude of the motion of the object on the target moving image is detected from the optical flow. You may make it judge that a moving image is what image | photographed the sport scenery. When such a determination is made, the same expansion process as when the shooting scene is set to “sports” can be performed.

  Further, for example, when the video signal analysis unit 34 analyzes the target video signal and finds that a person and a baseball bat are reflected on the target moving image, the target moving image is obtained by shooting a baseball batting scene. It can be judged that there is. If such a determination is made, the volume of the impulse sound that is estimated to be a striking sound is increased in the process of expansion in order to increase the playback volume of the striking sound and improve the force at the time of playback. Is possible.

<< Second Embodiment >>
A second embodiment of the present invention will be described. In the first embodiment described above, the sound signal is expanded in the process from collecting the sound signal and recording it on the recording medium 15, but the expansion process may be executed in the reproduction stage. . In the second embodiment, an imaging apparatus that executes decompression processing in the reproduction stage will be described. Since the overall configuration of the imaging apparatus according to the second embodiment is the same as that of FIG. 1, the imaging apparatus according to the second embodiment is also referred to as the imaging apparatus 1. The matters described in the first embodiment are applied to this embodiment as long as there is no contradiction.

  In the second embodiment, a signal obtained by encoding the target audio signal, which is the original audio signal, as well as a signal obtained by encoding the video signal of the target moving image is temporarily associated with the recording medium 15 and recorded. Is done. Thereafter, in response to an operation for instructing reproduction of the target moving image, a signal obtained by encoding the video signal of the target moving image is read from the recording medium 15 as a video signal stream, and the target audio signal is encoded as it is. The obtained signal is read out as an acoustic signal stream.

  FIG. 25 is a block diagram of a part related to generation of the extended acoustic signal according to the second embodiment. The sound source type analyzing unit 31, the sound signal extending unit 32, and the video signal analyzing unit 34 are the same as those in FIG. As described above, the sound source type analysis unit 31 and the acoustic signal expansion unit 32 may be abbreviated as the analysis unit 31 and the expansion unit 32, respectively. The analysis unit 31, the decompression unit 32, and the audio signal decoding unit 35 can be provided in the audio signal processing unit 14 of FIG. 1, and the video signal analysis unit 34 and the video signal decoding unit 36 are connected to the video signal processing of FIG. It can be provided in the part 12.

  The audio signal stream and the video signal stream read from the recording medium 15 are decoded by the audio signal decoding unit 35 and the video signal decoding unit 36, respectively, to generate the target audio signal and the target video signal. The target audio signal from the audio signal decoding unit 35 is sent to the analysis unit 31 and the expansion unit 32, and the target video signal from the video signal decoding unit 36 is sent to the video signal analysis unit 34. Similarly to the first embodiment, the analysis unit 31 and the video signal analysis unit 34 generate sound source type information and video analysis information based on the target audio signal and the target video signal, and send the information to the decompression unit 32.

  When the scene setting information described in the first embodiment is recorded on the recording medium 15, the scene setting information is sent from the recording medium 15 to the expansion unit 32. When the user inputs scene setting information during playback, the scene setting information input during playback may be given to the decompression unit 32. The decompression unit 32 is also given playback speed information. The reproduction speed information is information indicating the ratio between the shooting rate and the reproduction rate in the target moving image, and may be the same as the frame rate information described in the first embodiment.

  Now, as in the first embodiment, it is assumed that the shooting rate of the target moving image is 600 fps and the playback rate of the target moving image is 60 fps. Then, the decompression unit 32 follows the playback speed information, and based on all or part of the target sound signal, sound source type information, video analysis information, and scene setting information, the target sound for α seconds as in the first embodiment. An acoustic signal for (10 × α) seconds is generated from the signal as an extended acoustic signal.

  By sending the target video signal obtained by the decoding of the video signal decoding unit 36 to the display unit 16 at 60 fps, the target moving image is reproduced and displayed at 60 fps, and the expanded sound is synchronized with the reproduction of the target video signal. By transmitting the signal to the speaker 17, the extended acoustic signal synchronized with the reproduced video of the target moving image is reproduced as sound over (10 × α) seconds. For convenience of explanation, the case where the shooting rate and the reproduction rate are 600 fps and 60 fps, respectively, has been described. Of course, this is merely an example. Actually, for example, the shooting rate and the playback rate are 60 fps and 30 fps, respectively.

  Further, the decompression method may be changed based on a user instruction during playback. For example, the user can instruct that the target audio signal should be subjected to simple extension processing, and the contents of the instruction can be included in the scene setting information given to the extension unit 32. When such an instruction is given to the decompression unit 32, the decompression unit 32 performs simple decompression processing on the target acoustic signal from the acoustic signal decoding unit 35 without depending on the sound source type information and the video analysis information. Generate a stretched acoustic signal.

<< Deformation, etc. >>
The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. As modifications or annotations of the above-described embodiment, notes 1 to 3 are described below. The contents described in each comment can be arbitrarily combined as long as there is no contradiction.

[Note 1]
25, the analysis unit 31, the expansion unit 32, the video signal analysis unit 34, the audio signal decoding unit 35, the video signal decoding unit 36, and the display unit and the speaker equivalent to the display unit 16 and the speaker 17 of FIG. A playback device (not shown) may be configured separately from the imaging device 1. If the audio signal stream and the video signal stream from the recording medium 15 are given to such a reproducing apparatus, reproduction similar to that of the imaging apparatus 1 according to the second embodiment is realized on the reproducing apparatus.

  Note that the imaging device 1 according to the first embodiment has a function as a recording device that records video signals and audio signals, and the imaging device 1 according to the second embodiment reproduces video signals and audio signals. A function as a playback device is provided. An imaging device is a type of electronic device, and a recording device or a playback device is also a type of electronic device.

[Note 2]
The imaging apparatus 1 in FIG. 1 or the electronic device can be configured by hardware or a combination of hardware and software. When the imaging apparatus 1 or the electronic device is configured using software, a block diagram of a part realized by software represents a functional block diagram of the part. A function realized using software may be described as a program, and the function may be realized by executing the program on a program execution device (for example, a computer).

[Note 3]
For example, it can be considered as follows. An output acoustic signal generation unit that generates a decompressed acoustic signal as an output acoustic signal from a target acoustic signal as an input acoustic signal collected at the time of capturing a target moving image is formed including an analysis unit 31 and an expansion unit 32. (See FIG. 7 or FIG. 25). The acoustic signal processing device including the output acoustic signal generation unit can be considered to correspond to the acoustic signal processing unit 14, to be included in the acoustic signal processing unit 14, or to include the acoustic signal processing unit 14.

DESCRIPTION OF SYMBOLS 1 Imaging device 11 Imaging part 12 Video signal processing part 13 Microphone part 14 Acoustic signal processing part 31 Sound source type analysis part 32 Acoustic signal expansion part 33 Acoustic signal encoding part 34 Video signal analysis part

Claims (6)

An output acoustic signal generation unit that generates an output acoustic signal having a signal length longer than the input acoustic signal from an input acoustic signal picked up when the target moving image is captured at the first frame rate; An acoustic signal processing device,
The output acoustic signal is an acoustic signal to be reproduced as a sound together with the target moving image when the target moving image is reproduced at a second frame rate smaller than the first frame rate.
The output acoustic signal generation unit generates the output acoustic signal from the input acoustic signal according to a type of a sound source of the input acoustic signal. The output sound signal generation unit includes a sound source type analysis unit that analyzes a type of a sound source of the input sound signal based on the input sound signal, and the sound source of the input sound signal analyzed by the sound source type analysis unit The acoustic signal processing apparatus according to claim 1, wherein the output acoustic signal is generated from the input acoustic signal according to a type. The sound source type analysis unit determines whether or not a human voice is included in the sound source of the input sound signal based on the input sound signal,
The output acoustic signal generation unit changes a method of generating the output acoustic signal from the input acoustic signal according to whether or not a human voice is included in a sound source of the input acoustic signal. The acoustic signal processing apparatus according to claim 2. When the input sound signal includes sound signals from a plurality of different sound sources, the output sound signal generation unit uses the sound source type analysis unit to generate sound signals from the plurality of sound sources. Analyzing the type of the sound source of each separated acoustic signal while extracting it as the separated acoustic signal individually from the input acoustic signal, and then subjecting each separated acoustic signal to expansion processing according to the type of the sound source of each separated acoustic signal. The acoustic signal processing apparatus according to claim 2, wherein the output acoustic signal is generated by synthesizing the plurality of separated acoustic signals. The output sound signal generation unit generates the output sound signal from the input sound signal based not only on the analysis result by the sound source type analysis unit but also on the analysis result on the video signal of the target moving image. The acoustic signal processing device according to any one of claims 2 to 4. An electronic apparatus comprising the acoustic signal processing device according to any one of claims 1 to 5,
When shooting the target moving image at a first frame rate, generating the output acoustic signal from the input acoustic signal and recording the output acoustic signal on a recording medium; or
When the input sound signal is recorded on the recording medium and the target moving image is reproduced at the second frame rate, the output sound signal is generated from the recorded input sound signal, and the target moving image is generated. And an electronic apparatus that reproduces the output acoustic signal.

Images