JP2020072311A - Information acquisition device, information acquisition method, information acquisition program, and information acquisition system - Google Patents

Abstract

To obtain a preferable sound corresponding to a captured image by selecting and adjusting a sound from an external sound collecting device.SOLUTION: An information acquisition device includes a feature extraction unit 11g that acquires a first sound collected by a built-in sound collecting device 13 that is built in an imaging device 10 and that collects an ambient sound of the imaging device, and extracts the characteristic of the acquired first sound, a sound acquisition unit 21b that acquires a third sound by performing at least one of selection and adjustment of a plurality of second sounds collected by a plurality of microphones having different sensitivity distribution directions on the basis of the characteristics of the first sound extracted by the feature extraction unit, and a synchronization processing unit 11h that synchronizes the third sound acquired by the sound acquisition unit with an image of a subject imaged by the imaging device.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information acquisition device, an information acquisition method, an information acquisition program, and an information acquisition system that acquire video and audio from an imaging device and a sound collection device.

  In recent years, many mobile devices (photographing devices) with a photographing function such as digital cameras have a moving image photographing function as well as a still image. The image capturing device is equipped with a built-in microphone that picks up surrounding sounds, and there are some that can record AV data including images and sounds when shooting a moving image.

  Further, some photographing devices have a terminal to which an external microphone can be attached, and a device capable of recording an image obtained by shooting a moving image and a sound obtained by the external microphone has been commercialized. For example, when an external microphone can be arranged at a position distant from the photographing device, by arranging the external microphone in the vicinity of the subject that is the sound collection target, sound with good S / N can be output from the external microphone. It is also possible to obtain.

JP, 2005-151471, A

  However, for example, when photographing a wild bird, the user carrying the photographing device can move to a position relatively far from the subject and can photograph the subject satisfactorily, while arranging it near the subject. The movement of the external microphone may be restricted, and it is not always possible to acquire the sound suitable for the image acquired by the photographing device with the external microphone.

  In addition, in patent document 1, there is known a TV conference system using a plurality of microphones and cameras, which accurately selects a speaker. However, this system has a voiceprint authentication unit that authenticates conference participants who have registered voiceprints, an image pickup adjustment unit that controls the television camera device so as to optimally capture a speaker, and the like, and thus the device scale is large.

  The present invention, by selecting or adjusting the sound from the external sound collecting device based on the characteristics of the sound acquired by the built-in sound collecting device, provides an image captured by the image capturing device and a preferable sound corresponding to the image. It is an object of the present invention to provide an information acquisition device, an information acquisition method, an information acquisition program, and an information acquisition system capable of acquiring information.

  An information acquisition device according to an aspect of the present invention acquires a first voice that is picked up by a built-in sound pickup device that is built in an image pickup device and picks up ambient sound of the image pickup device, and the acquired first voice. Of a plurality of second voices picked up by a plurality of microphones having different sensitivity distribution directions based on the feature extracting unit for extracting the feature of the first voice and the feature of the first voice extracted by the feature extracting unit. An audio acquisition unit that performs at least one of the operations to acquire a third audio, and a synchronization processing unit that synchronizes the third audio acquired by the audio acquisition unit with the image of the subject obtained by the imaging device. ..

  An information acquisition method according to an aspect of the present invention is an information acquisition method in an information acquisition device including a feature extraction unit, a voice acquisition unit, and a synchronization processing unit, wherein the feature extraction unit is built in an imaging device to perform the imaging. The first sound picked up by the built-in sound collecting device that picks up the ambient sound of the device is extracted, the characteristic of the acquired first sound is extracted, and the sound acquisition unit extracts the first sound. On the basis of the characteristics of, the plurality of microphones having different sensitivity distribution directions perform at least one of selection and adjustment with respect to the plurality of second voices to acquire the third voice, and the synchronization processing unit acquires the third voice. The third sound is synchronized with the image of the subject obtained by the image pickup device.

  The information acquisition program according to one aspect of the present invention acquires the first sound picked up by a built-in sound collecting device, which is built in the image pickup device and picks up ambient sound of the image pickup device, in the computer, and is acquired. A feature of the first voice is extracted, and based on the extracted feature of the first voice, at least one of selection and adjustment of the plurality of second voices picked up by the plurality of microphones having different sensitivity distribution directions is performed. A third sound is acquired, and a procedure for synchronizing the acquired third sound with the image of the subject obtained by the imaging device capturing the third sound is executed.

  An information acquisition system according to one aspect of the present invention includes an imaging device that includes a built-in sound pickup device that picks up ambient sound, and a feature extraction unit that extracts a feature of the first sound picked up by the built-in sound pickup device. A plurality of microphones having different sensitivity distribution directions, and a plurality of second sounds picked up by the plurality of microphones are selected based on the characteristics of the first sound extracted by the characteristic extraction unit of the imaging device, and An external sound collecting device that performs at least one of the adjustments to acquire the third sound, and a synchronization process that synchronizes the third sound acquired by the external sound collecting device with the image of the subject obtained by the imaging device. And a reproducing device for performing.

  According to the present invention, by selecting or adjusting the sound from the external sound collecting device based on the characteristics of the sound acquired by the built-in sound collecting device, the image taken by the photographing device and the sound corresponding to the image are picked up. It has the effect that and can be obtained.

The block diagram which shows the information acquisition apparatus which concerns on the 1st Embodiment of this invention. Explanatory drawing which shows an example of the external appearance of the camera 10 and the external sound collection device 20. Explanatory drawing for demonstrating the state of photography. Explanatory drawing for demonstrating the state of photography. Explanatory drawing which shows the positional relationship of the to-be-photographed object of the camera 10 and the external sound collection device 20 at the time of imaging. 6 is a flowchart for explaining the operation of the camera 10. The flowchart for demonstrating operation | movement of the external sound collection device 20. The block diagram which shows the information acquisition apparatus which concerns on the 2nd Embodiment of this invention. 6 is a flowchart for explaining the operation of the camera 50. The block diagram which shows the information acquisition apparatus which concerns on the 3rd Embodiment of this invention. Explanatory drawing for demonstrating the external appearance of the camera 60 and the recorder 70. Explanatory drawing for demonstrating the difference between the angle from ST sound collection part 13 to a subject, and the angle from ST sound collection part 72 to a subject. 6 is a flowchart for explaining the operation of the camera 60. 6 is a flowchart for explaining the operation of the recorder 70. 6 is a flowchart for explaining the operation of the playback device 80.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing an information acquisition device according to the first embodiment of the present invention. FIG. 2 is an explanatory diagram showing an example of the external appearance of the camera 10 and the external sound collecting device 20. The present embodiment employs an image pickup device that is movable and has a built-in sound pickup device and is capable of shooting a moving image of a subject. And the image acquired by the image pickup device by selecting or adjusting the sound from the external sound collecting device based on the comparison of the characteristics of the sounds collected by the built-in sound collecting device and the external sound collecting device. It makes it possible to obtain the appropriate voice. In addition, in the present embodiment, the sound suitable for the video is, for example, the sound generated by the main subject with a high S / N ratio.

  The information acquisition device according to the present embodiment may be configured in the image pickup device, may be configured in the external sound pickup device, or may be configured in a distributed manner in the image pickup device and the external sound pickup device, Further, it may be configured as a device independent of these devices. It should be noted that FIG. 1 shows an example in which the information acquisition devices are distributed and configured in the imaging device and the external sound collecting device.

  First, the external appearance of the camera 10 and the external sound pickup device 20, which are image pickup devices, will be described with reference to FIG.

  The camera 10 shown in FIG. 2 has a housing 10a in which each circuit of FIG. 1 is housed and a lens barrel 12b in which an optical system 12a described later is housed. A shutter button 15a that constitutes an operation unit 15 described later is provided on the upper surface of the housing 10a.

  Further, the external sound pickup device 20 shown in FIG. 2 has a housing 20a in which each circuit of FIG. 1 is housed. The housing 20a is formed in a cylindrical shape, and the sound pickup portions of a plurality of microphones 22a that form a multi-sound pickup portion 22, which will be described later, are provided so as to project from the peripheral surface toward the surroundings. For example, twelve microphones 22a are arranged every 30 degrees in the normal direction of the side surface of the housing 20a. The multi-sound pickup unit here displays a plurality of microphones with different distribution directions of the sound that can be picked up at the time of sound pickup, but it is sufficient if the number of the microphones is plural. The above devices may be combined and configured.

  In FIG. 1, a control unit 11 is provided in a camera 10 that constitutes the image pickup apparatus. The control unit 11 may be configured by a processor using a CPU, an FPGA, or the like, operate according to a program stored in a memory (not shown) to control each unit, or may be a function of an electronic circuit of hardware. Some or all may be realized.

  The camera 10 includes an image capturing unit 12 and an ST sound collecting unit 13. The image pickup unit 12 has an optical system 12a and an image pickup device (not shown). The optical system 12a includes a lens, a diaphragm, and the like (not shown) for zooming and focusing. The optical system 12a includes a zoom (variable magnification) mechanism, a focus and a diaphragm mechanism (not shown) that drives these lenses. The image pickup device is composed of a CCD, a CMOS sensor or the like, and the optical system 12a guides the subject optical image to the image pickup surface of the image pickup device. The image pickup device photoelectrically converts the subject optical image to obtain a picked-up image (image pickup signal) of the subject.

  The photographing control unit 11a included in the control unit 11 is capable of controlling the zoom mechanism, the focus mechanism, and the aperture mechanism of the optical system 12a to adjust the zoom, aperture, and focus. The focus / angle-of-view information unit 11c is configured to acquire information regarding zoom, aperture, and focus from the optical system 12a and output the information to the photographing control unit 11a. By this feedback, the imaging control unit 11a can set the zoom, aperture, and focus to desired setting values. The image capturing unit 12 is controlled by the image capturing control unit 11a to perform image capturing, and outputs an image capturing signal of a captured image (moving image and still image) to the control unit 11.

  A sound collection control and processing unit 11e is configured in the control unit 11, and the sound collection control and processing unit 11e controls the ST sound collection unit 13. The ST sound collecting unit 13 as a built-in sound collecting device is configured by a stereo microphone or the like, and is controlled by the sound collecting control and processing unit 11e to collect the sound around the camera 10 and acquire a sound signal. The acquired voice (hereinafter, also referred to as internal voice or first voice) can be output to the control unit 11. The ST sound collecting unit 13 has a peak sensitivity in the shooting direction of the camera 10, that is, in the optical axis direction of the optical system 12a.

  The camera 10 is provided with an operation unit 15. The operation unit 15 includes a release button, a function button, various switches for shooting mode setting, parameter operation, etc., a dial, a ring member, etc. (not shown), and outputs an operation signal based on a user operation to the control unit 11. The control unit 11 controls each unit based on an operation signal from the operation unit 15.

  The control unit 11 captures the captured image (moving image and still image) from the image capturing unit 12. The image processing unit 11b of the control unit 11 performs predetermined signal processing, for example, color adjustment processing, matrix conversion processing, noise removal processing, and various other signal processing on the captured image.

  The camera 10 is provided with a display unit 16, and the display unit 16 has a display screen such as an LCD (liquid crystal display device). This display screen is provided, for example, on the back surface of the housing of the camera 10. The control unit 11 is configured to display the captured image signal-processed by the image processing unit 11b on the display unit 16. The control unit 11 can also display various menu displays, warning displays, etc. of the camera 10 on the display unit 16.

  The camera 10 is provided with communication units 18a and 18b. The communication units 18a and 18b are controlled by the control unit 11 and can transmit and receive information to and from the external sound collecting device 20. The communication unit 18a can perform short-range wireless communication such as Bluetooth (registered trademark), and the communication unit 18b can perform wireless LAN communication such as Wi-Fi (registered trademark). Note that the communication units 18a and 18b are not limited to Bluetooth and Wi-Fi, but can employ communication in various communication systems. The control unit 11 can receive an audio signal from the external sound collection device 20 via the communication unit 18a or 18b.

  The camera 10 is provided with a recording unit 17. The recording unit 17 is composed of a predetermined recording medium and can record the information given from the control unit 11 and output the recorded information to the control unit 11. A card interface, for example, can be used as the recording unit 17, and the recording unit 17 can record image data on a recording medium such as a memory card.

  In the present embodiment, the recording unit 17 has a sound pickup image / sound recording unit 17a, a cooperation information unit 17b, and an external sound recording unit 17c. The control unit 11 can compress the captured image after the signal processing and give the compressed image to the recording unit 17 for recording. This image is recorded in the picked-up image audio recording unit 17a. The camera 10 is provided with a clock unit 19, and the control unit 11 uses the time information from the clock unit 19 to associate the moving image acquired by the imaging unit 12 with the sound acquired by the ST sound collecting unit 13. It can be recorded in the collected sound image sound recording unit 17a.

  Information related to communication with the external sound collecting device 20 is recorded in the cooperation information unit 17b, and the control unit 11 controls the communication units 18a and 18b based on the information read from the cooperation information unit 17b. Thus, information can be exchanged with the external sound collecting device 20 by communication. As a result of communication with the external sound collecting device 20, the control unit 11 can receive an audio signal from the external sound collecting device 20. The control unit 11 is capable of giving the received voice as an external voice (also referred to as a third voice) to the external voice recording unit 17c to record the voice.

  In the present embodiment, the control unit 11 includes a voice acquisition unit 11f. The voice acquisition unit 11f has a feature extraction unit 11g for designating a voice to be acquired as an external voice. The feature extraction unit 11g extracts the voice feature of the internal voice acquired by the ST sound collection unit 13. The voice acquiring unit 11f transmits the voice feature acquired by the feature extracting unit 11g to the external sound collecting device 20 via the communication unit 18a or 18b as information for designating the voice to be acquired as the external voice. Is becoming It should be noted that the voice feature may be determined by, for example, analyzing the frequency range, the state of change in frequency, the state of change in strength (voice amplitude), and the like, and the feature extraction unit 11g determines these states as known. Various determination methods may be adopted and extracted.

  In addition, a method of extracting a specific sound component from a speech input using an inference engine obtained by machine learning is also conceivable in order to make a comprehensive determination of these. For this purpose, it is sufficient to create a specific amount of teacher data by using sound information of a specific time width and voice extracted therefrom as correct answer data, and perform machine learning so that these can be discriminated. Furthermore, there is also a method of performing analysis in combination with changes in images acquired in synchronization.

  The external sound collecting device 20 is provided with a control unit 21. The control unit 21 may be configured by a processor using a CPU or the like and operate according to a program stored in a memory (not shown) to control each unit, or a part of a function of an electronic circuit of hardware. Alternatively, all of them may be realized. The external sound collecting device 20 has communication units 26a and 26b. The communication units 26a and 26b are controlled by the control unit 21 so that information can be transmitted and received to and from the camera 10. The communication unit 26a can perform short-range wireless communication such as Bluetooth (registered trademark), and the communication unit 26b can perform wireless LAN communication such as Wi-Fi (registered trademark). Note that the communication units 26a and 26b are not limited to Bluetooth and Wi-Fi, but can employ communication in various communication systems. The control unit 21 can receive audio feature information from the camera 10 and transmit an audio signal to the camera 10 via the communication unit 26a or 26b.

  The external sound pickup device 20 is provided with a multi-sound pickup unit 22, and the multi-sound pickup unit 22 is composed of, for example, a plurality of microphones (not shown), and each microphone has a predetermined directional characteristic in different directions. Sensitivity distribution). That is, the multi-sound collecting unit 22 is adapted to acquire a plurality of sounds (hereinafter, also referred to as an external sound collection sound or a second sound) picked up by a plurality of microphones having sensitivity peaks in mutually different directions. There is. The control unit 21 is configured with a sound pickup control unit 21a, and the sound pickup control unit 21a can control the sound pickup of the multi sound pickup unit 22.

  The external sound pickup device 20 is provided with an operation unit 23. The operation unit 23 includes various switches, dials, ring members and the like (not shown) for recording mode setting, parameter operation, etc., and outputs an operation signal based on a user operation to the control unit 21. The control unit 21 controls each unit based on an operation signal from the operation unit 23. Moreover, when the control information is given from the control unit 11 of the camera 10 via the communication units 26a and 26b, the control unit 21 may control each unit based on this control information. In this case, the control unit 11 of the camera 10 can control recording in the external sound pickup device 20.

  The external sound collecting device 20 is provided with a recording unit 25. The recording unit 25 is configured by a predetermined recording medium, can record the information given from the control unit 21, and can output the recorded information to the control unit 21. A card interface, for example, can be used as the recording unit 25, and the recording unit 25 can record image data on a recording medium such as a memory card.

  The recording unit 25 has a voice recording unit 25a and a cooperation information unit 25b. The control unit 21 can give the externally collected sound after the signal processing to the sound recording unit 25a to record the sound. The external sound pickup device 20 is provided with a clock unit 24, and the control unit 21 uses the time information from the clock unit 24 to add time information to the external sound pickup sound acquired by the multi sound pickup unit 22. Can be recorded in the voice recording unit 25a.

  In the present embodiment, the sound acquisition unit 21b is provided to determine the third sound to be transmitted to the camera 10. The voice acquisition unit 21b has a feature extraction unit 21c. The feature extraction unit 21c has the same configuration as the feature extraction unit 11g, and extracts each of the voice features of the plurality of external sound pickup sounds (second sound) acquired by the multi sound pickup unit 22. The voice acquisition unit 21b receives information for designating a voice to be transmitted as external voice (third voice) from the camera 10 via the communication unit 26a or 26b, that is, the internal voice (first voice acquired by the camera 10). (Voice) information of voice characteristics is given. The voice acquisition unit 21b selects the external collected voice to be output as the external voice by comparing the voice feature received from the camera 10 and the voice feature of each external collected voice extracted by the feature extraction unit 21c. ..

  For example, the voice acquisition unit 21b selects an external picked-up voice having a voice feature having the highest similarity to the voice feature of the internal voice, and the selected external picked-up voice is used as the external voice (third voice) together with the time information. The data is transmitted to the camera 10 via the communication unit 26a or 26b. In addition, when there are a plurality of external sound pickup sounds having a sound feature whose similarity is higher than a predetermined threshold value, the sound acquisition unit 21b has the highest S / N ratio among these external sound pickup sounds. The voice may be selected as the external voice.

  The voice acquisition unit 21b may be capable of performing voice processing such as individually adjusting the gain of a plurality of voice signals picked up by the multi-voice pickup unit 22 or synthesizing the voice signals at a predetermined ratio. The sound acquisition unit 21b may be configured to select one or more externally collected sounds, adjust the selected externally collected sounds, and acquire the external sound (third sound). Note that stereo sound may be acquired as the external sound.

  A synchronization processing unit 11h is configured in the control unit 11 of the camera 10. The synchronization processing unit 11h synchronizes the external sound acquired by the sound acquisition unit 11f with the image of the subject acquired by the imaging unit 12 and records the external sound in the external sound recording unit 17c. For example, the synchronization processing unit 11h synchronizes the image of the subject with the external sound by comparing the waveform of the audio signal picked up at the same time as the video signal acquired by the imaging unit 12 with the waveform of the external sound. It may be like this.

  By thus providing the reference signal (by the internal audio recording unit built in the camera), the association such as synchronization of information from the outside can be simplified as the system configuration. This is because the image capture and audio acquisition are managed and recorded by the clock signal in the same device, so you can refer to what is already synchronized in the camera as the correct situation, or based on this standard. This means that operations such as eliminating the sound velocity delay due to the distance between the objects can be performed easily.

  Next, the operation of the embodiment thus configured will be described with reference to FIGS. 3 to 7. FIG. 3 and FIG. 4 are explanatory diagrams for explaining the shooting state, and FIG. 5 is an explanatory diagram showing the positional relationship between the camera 10 and the external sound collecting device 20 and the subject at the time of shooting. 6 is a flow chart for explaining the operation of the camera 10, and FIG. 7 is a flow chart for explaining the operation of the external sound collecting device 20.

  In the example of FIGS. 3 and 4, the camera 10 is provided with the display screen 16a of the display unit 16 on the back surface of the housing 10a. The user, for example, grips the housing 10a with his / her hand and, while watching the display screen 16a, captures the bird 41, which is a subject, in the visual field range, and presses the shutter button 15a to perform shooting. 3 and 4, the shooting range of the camera 10 is shown surrounded by a frame.

  In the example of FIG. 3, the bird 41 stays on the branch of the tree 31a, and in the example of FIG. 4, the bird 41 stays on the branch of the tree 31b next to the tree 31a. Grass 32 grows on the ground between the trees 31a and 31b and the camera 10. The external sound pickup device 20 is arranged at a position relatively close to the trees 31a and 31b. For example, when shooting a wild bird, it is conceivable to install the external sound pickup device 20 in advance near a tree where the wild bird is likely to stay. On the other hand, the user carrying the camera 10 shoots at a position relatively distant from the wild bird in order to prevent the wild bird from escaping and to avoid obstacles.

  Therefore, if the sound of the bird 41 is picked up, the external sound picked up by each microphone 22a of the multi sound pick-up unit 22 is more than the sound picked up by the ST sound pick-up unit 13 (first sound) built in the camera 10. It is considered that better S / N can be obtained with the sound voice (second voice). It should be noted that by using a telephoto lens etc. for video, it is possible to obtain a sufficiently high quality image even from a position relatively distant from the subject, but with regard to sound, noise will occur as the distance from the subject increases. However, the sound quality of the target sound is deteriorated. Therefore, it is possible to acquire a voice with less noise by collecting sound with a microphone located closer to the object.

  When the power is turned on, the control unit 11 of the camera 10 determines whether or not the imaging mode is instructed in step S1 of FIG. When the shooting mode is not instructed, the control unit 11 sets the specified mode, for example, the cooperation mode for setting and transmitting / receiving for cooperation with the external sound collecting device 20, or the reproduction mode for reproducing recorded images. Move to.

  When the imaging mode is instructed, the control unit 11 determines in next step S2 whether or not the cooperation with the external sound pickup device is designated. When the cooperation is not designated, the control unit 11 moves the process to step S6 and determines whether the recording start operation is performed or whether the recording is being performed. When the cooperation is designated, the control unit 11 performs the image pickup and the sound recording for designating the external sound before the actual image pickup and the sound recording by the user operation in step S3.

  In the next step S4, the feature extraction unit 11g of the control unit 11 extracts the voice feature of the internal voice (first voice) from the ST sound pickup unit 13 and transmits the extracted voice feature via the communication units 18a and 18b. The sound is transmitted to the external sound collecting device 20 (step S5), and the process proceeds to step S6. It should be noted that the ST sound collecting unit 13 has a peak sensitivity in the shooting direction, and the internal sound acquired by the ST sound collecting unit 13 is considered to have the sound characteristics of the bark of the bird 41 that is the subject.

  On the other hand, when the power is turned on, the control unit 21 of the external sound collection device 20 determines in step S21 of FIG. 7 whether the cooperation with the camera 10 is set. When the cooperation is not set, the control unit 21 shifts the processing to step S25 and determines whether or not sound collection is instructed. The control unit 21 may determine that the sound collection is instructed by transmitting the information indicating the start of recording from the camera 10. When the sound collection is not instructed, the control unit 21 shifts the processing to step S31, and performs other processing, for example, a cooperation mode for setting or transmitting / receiving for cooperation, and reproducing recorded sound. Playback mode etc. are executed. In the cooperation mode, the audio features acquired by the feature extraction unit 11g of the camera 10 are received by the communication units 26a and 26b and recorded in the recording unit 25.

  When the cooperation is set, the control unit 21 shifts the processing from step S21 to step S22 in order to determine the external sound (third sound) to be transmitted to the camera 10, and the multi sound collecting unit 22. Sound is picked up by all microphones of. When the control unit 21 acquires the external sound pickup voice (second voice) from all the microphones of the multi sound pickup unit 22 in step S23, the feature extraction unit 21c extracts a voice feature. The voice acquisition unit 21b compares the voice feature of each external collected voice (second voice) with the voice feature of the internal voice (first voice) read from the recording unit 25.

  The voice acquisition unit 21b determines the microphone that picks up the voice output as the external voice among all the microphones of the multi-voice pickup unit 22 that acquires the external voice pickup by comparing the voice characteristics (step S24). For example, the voice acquisition unit 21b calculates the similarity between the voice features of the first voice and each of the second voices, and the S / N is the second voice having the voice feature of which the similarity is higher than a predetermined threshold. Is selected, and the selected microphone is determined as the microphone for collecting external sound.

  This S / N ratio determination may need to determine which is the signal (S) and which is the noise (N), but this can be done even by using the feature determination technique described above. Of course, it may be possible to infer from the relationship between the information of the image of the object to be photographed and the sound uttered by the object. For example, the chirping of birds and the voice of a person correlate with the movement of the mouth and throat, so the feature pattern is detected from the image by color, shading, and shape, and the pattern of its change matches the pattern of voice change. May be selected as the signal (S). In this case, the other component is noise (N). In addition, the image of the target can be understood from the image because the photographer can see what the target is interested in (the sound is easy to spread, so it is difficult to aim, but the image goes straight with light, so the target is clear) Characters (shape, color distribution, and motion characteristics) are determined using an image dictionary, and for humans, a voice component that matches the human voice's characteristics is extracted as a signal (S), and the others are noise. There is also a technical solution such as (N). If the bird is detected, the voice of the bird is detected, if the cat is detected, the voice of the cat is detected, and if an instrument is detected, the component of the sound obtained at that time is the characteristic of the sound related to the image. It is possible to easily discriminate by preparing a dictionary or database that lists, or an inference model obtained by machine learning. These can be composed of a recording unit and a calculation unit. It is also possible to simply select a specific frequency or a specific change pattern such as wind sound, air conditioning, or crowdedness to determine noise (N), and select a sound component having other characteristics as the signal (S). In some cases, both the flow of water and the voice of the bird are important, but in such a case, it is possible to devise a method to take both of them cleanly. The determination when a plurality of voices are important may use an inference model obtained by machine learning. In this case, an annotation may be performed so that a plurality of sound components are selected as teacher data during machine learning. Further, the microphones of the multi-sound pickup unit may be selected from a plurality of microphones instead of selecting only one microphone. Also, the sound of one microphone may be processed using the sound collection results of a plurality of microphones. If you want to emphasize the stereo effect, you can select the same microphone for each of the left and right voices, and use the built-in microphones and the object position on the screen so that the voices obtained with multiple microphones give a natural stereo effect. Voice processing may be performed using information such as.

  FIG. 5 shows the positional relationship among the camera 10, the multi sound collecting unit 22, and the subject (bird 41) in the examples of FIGS. In FIG. 5, the distance from the camera 10 to the multi-sound pickup unit 22 is about Dm, and the distance from the camera 10 to the bird 41, which is a subject, is about D0m. FIG. 5 shows an example in which the microphones 22a of the external sound pickup device 20 are configured by 12 microphones M1 to M12 which are sequentially arranged at intervals of 30 degrees, and the microphones M2, M7, M11 are indicated by broken lines. The sensitivity peak direction is shown, and the solid line shows the sensitivity peak direction DM1 of the microphone M1 and the sensitivity peak direction DM2 of the microphone M12.

  The sensitivity peak direction DM1 of the microphone M1 matches the direction toward the bird 41 in FIG. 3, and the sensitivity peak direction DM2 of the microphone M12 matches the direction toward the bird 41 in FIG. Therefore, in the example of FIG. 3, it is considered that the voice feature of the externally collected voice by the microphone M1 is most similar to the voice feature of the internal voice, and in the example of FIG. 4, the voice feature of the externally collected voice by the microphone M12 is It is considered to be most similar to the voice feature of internal voice.

  However, when there is no sound source that generates a sound other than the bird 41, the similarity between the voice characteristics of the externally collected voice and the voice characteristics of the internal voice of the plurality of microphones is larger than a predetermined threshold value. It can be a value. Also in this case, the peak sensitivity direction is different because the sensitivity distribution directions of the microphones are different, and in the case where the sound of the bird 41 is picked up by the microphones M1 to M12, the microphone having the highest S / N is shown in FIG. In this case, it is considered to be the microphone M1, and in the case of FIG. 4, it is considered to be the microphone M12. As a result, in the example of FIG. 3, the microphone M1 is determined as the microphone for collecting external voice, and in the example of FIG. 4, the microphone M12 is determined as the microphone for collecting external voice.

  When the sound pickup is instructed, the multi sound pickup unit 22 shifts the processing from step S25 to step S26 and picks up the sound. The control unit 21 determines whether or not the cooperation with the camera 10 is specified, and when the cooperation is specified, the sound from the microphone selected in step S24 is transmitted to the camera 10 as the third sound. Yes (step S28). The control unit 21 determines the end of the sound collection in step S29, and repeats the processing of steps S26 to S29 until the end operation is performed. When the end of sound collection is determined, the recorded voice is converted into a file and the process returns to step S21.

  On the other hand, it is assumed that the user operates the operation unit 15 of the camera 10 to start capturing an image of a subject. When the control unit 11 determines that the recording start is instructed in step S6, the control unit 11 controls the image capturing unit 12 to capture an image of the subject and controls the ST sound collecting unit 13 to collect ambient sound (step S7). ). The control unit 11 starts recording the picked-up image and the collected sound in the collected-sound image sound recording unit 17a of the recording unit 17.

  In step S8, the control unit 11 determines whether or not cooperation with the external sound collection device 20 is being performed. When the cooperation is not performed, the control unit 11 determines whether or not the shooting end operation is performed in step S12. On the other hand, when the cooperation is performed, the control unit 11 receives the external voice transmitted from the external sound collecting device 20 in step S9, and acquires the internal voice in step S10. The synchronization processing unit 11h of the control unit 11 synchronizes the external sound with the video obtained from the imaging unit 12 by comparing the received external sound with the internal sound, and records the external sound in the external sound recording unit 17c (step S11), the process proceeds to step S12.

  The control unit 11 determines whether or not the shooting end operation is performed in step S12, and repeats steps S1, S2, and S6 to S11 until the shooting end operation is performed. When the shooting end operation is performed, the control unit 11 converts the recorded image and sound into a file (step S13), and returns the process to step S1.

  6 and 7, an example in which the external sound is received from the external sound collecting device 20 at the same time as the image pickup by the camera 10 and the sound collection of the internal sound, and the recording is performed while synchronizing the image and the external sound. However, even if the external sound pickup device 20 records the external sound and transmits the filed external sound to the camera 10 after capturing the image, the video and the external sound are recorded in a synchronized state. Good.

  As described above, in the present embodiment, an external sound pickup device capable of picking up sound with a predetermined directional characteristic in each of a plurality of directions is adopted, and the sound collected by the built-in sound pickup device and the external sound pickup device By selecting or adjusting the sound from the external sound pickup device based on the comparison of the features, the image acquired by the image pickup device and the sound suitable for the image are acquired, and the sound suitable for the image acquired by the image pickup device is acquired. Can be recorded in synchronization with the video. As a result, it is possible to automatically obtain an image in which the optimum sound is synthesized without performing the extremely complicated work of synthesizing the image and the sound suitable for the image by the editing work after the imaging.

  Note that, as described above, the information acquisition device may be configured in either the camera 10 or the external sound collection device 20, and may be configured, for example, only in the camera 10. In this case, as the external sound pickup device 20, a general microphone device that includes only a plurality of microphones having different peak sensitivity directions and a communication unit that transfers the sound picked up by each microphone to the camera 10 is adopted. You can

(Second embodiment)
FIG. 8 is a block diagram showing an information acquisition device according to the second embodiment of the present invention. In FIG. 8, the same components as those in FIG. 1 are designated by the same reference numerals and the description thereof will be omitted. In the first embodiment, the external sound pickup device is arranged in the vicinity of the subject, on the assumption that the voice feature of the voice emitted by the subject can be extracted from the voice acquired by the ST sound pickup unit 13 which is the built-in sound pickup device. This makes it possible to acquire external sound with good S / N emitted by the subject. However, since the ST sound collecting unit 13 and the subject are relatively distant from each other, it is conceivable that the sound feature of the subject cannot be reliably extracted by the sound collected by the ST sound collecting unit 13. Therefore, in the present embodiment, it is possible to reliably extract the voice feature generated by the subject by using the database in which the image feature and the voice feature are recorded. In the present embodiment, the camera 50 differs from the camera 10 of FIG. 1 in that an image feature extraction unit 14 and an image / sound (database) DB unit 17d are added.

  In FIG. 8, the image feature extraction unit 14 is configured by a processor or the like, and is configured to extract the image feature of the image captured by the imaging unit 12 and output the image feature to the control unit 11. The recording unit 17 is provided with an image / sound database (DB) unit 17d. The image and sound DB unit 17d stores image characteristics of various objects and sound characteristics of various objects, that is, information of sound characteristics of environmental sounds.

  The audio acquisition unit 11f determines the type of the main subject imaged by the imaging unit 12 by comparing the image features extracted by the image feature extraction unit 14 with the image features stored in the image audio DB unit 17d. .. The sound acquisition unit 11f reads out the determined sound feature generated by the main subject from the image sound DB unit 17d, and compares the sound feature of the internal sound acquired by the feature extraction unit 11g with the external sound collection device 20. It is designed to generate information on voice characteristics.

  For example, when the main subject is a bird, the sound acquisition unit 11f determines the type of bird from the image features of the image and sound DB unit 17d, and determines the sound features of the squeal of the determined type of bird in the image and sound DB unit 17d. Read from. The voice acquisition unit 11f removes noise components from the collected internal voice by using the read voice feature, and generates voice feature information to be designated to the external sound pickup device 20.

  In addition, for example, when the main subject is a musical instrument, the audio acquisition unit 11f determines the type of musical instrument from the image characteristics of the image / audio DB unit 17d, and determines the audio characteristic of the sound of the determined type of musical instrument in the image / audio DB. It is read from the section 17d. The voice acquisition unit 11f removes noise components from the collected internal voice by using the read voice feature, and generates voice feature information to be designated to the external sound pickup device 20. Similarly, the voice acquisition unit 11f can also extract voice features of a desired person's voice from a large number of people, for example.

  Although an example of determining the object based on the image features extracted by the image feature extraction unit 14 has been described with reference to FIG. 8, the type of sound to be extracted may be specified by the user's input operation. Good.

  Next, the operation of the embodiment configured as described above will be described with reference to the flowchart of FIG. FIG. 9 is a flow chart for explaining the operation of the camera 50. In FIG. 9, the same steps as those in FIG. The flow of FIG. 9 differs from that of FIG. 6 in that step S41 is adopted instead of step S4.

  In step S41, the voice acquisition unit 11f uses the image / voice DB unit 17d to extract the voice feature. For example, in the example of FIG. 3 and FIG. 4, the grass 32 is blown by the wind, which causes a "buzzing" sound, and this sound is louder than the sound of the bird 41, which is the subject, and the ST sound collecting unit 13 May be picked up. Therefore, the sound acquisition unit 11f specifies the main subject from the information stored in the image sound DB unit 17d based on the image features extracted by the image feature extraction unit 14, and determines the sound features of the identified main subject. It is read from the image / audio DB unit 17d. The voice acquisition unit 11f determines the noise component based on the read voice feature and the voice feature of the collected internal voice, and extracts the voice feature of the voice emitted from the main subject. The voice acquisition unit 11f transmits the extracted voice feature information to the external sound collection device 20 via the communication units 18a and 18b.

  Other actions are similar to those of the first embodiment.

  As described above, also in this embodiment, the same effect as that of the first embodiment can be obtained. Furthermore, in the present embodiment, as the information for selecting the external sound, not only the sound characteristic of the internal sound but also the image characteristic information is used, and the information other than the sound emitted by the main subject as the target of the internal sound is used. Even if the sound is included, the sound feature based on the sound emitted by the main subject can be surely extracted and used for the selection of the external sound.

  In each of the above embodiments, the example in which the sound picked up by one microphone is recorded in the camera 10 as the external sound is described. However, as the external sound, a plurality of sounds, for example, a stereo sound is selected and recorded. You may be allowed to do so.

(Third Embodiment)
FIG. 10 is a block diagram showing an information acquisition device according to the third embodiment of the present invention. 10, the same components as those in FIG. 1 are designated by the same reference numerals and the description thereof will be omitted. FIG. 11 is an explanatory diagram showing an example of the external appearance of the camera 60 and the recorder 70.

  As described above, the external sound pickup device is fixedly disposed, for example, near the subject, and the camera is carried by the user and is movably disposed at a position relatively distant from the subject. Therefore, the camera and the external sound pickup device may have different distances and directions with respect to the subject, and it may not always be possible for the external sound pickup device to obtain sound suitable for the image obtained by the camera. For example, even if the bird moves from right to left on the screen, the movement of the bird may not be felt by the external sound. Therefore, in the present embodiment, as the sound suitable for the video, the sound generated by the main subject is a sound in which the position in the image of the main subject in the video and the sound image localization of the sound emitted by the main subject match as much as possible. , For example, is adapted to acquire a stereo sound included with a S / N as high as possible.

  Also in the present embodiment, the information acquisition device may be configured in the image pickup device, in the external sound pickup device, or in the image pickup device and the external sound pickup device in a distributed manner. Further, they may be configured in a device independent of these devices. It should be noted that FIG. 10 shows an example in which the information acquisition device is distributed and configured in the camera 60, the recorder 70 and the reproduction device 80 that configure the external sound pickup device.

  First, the external appearance of the camera 60 as an image pickup device and the recorder 70 as an external sound pickup device will be described with reference to FIG.

  A camera 60 shown in FIG. 11 has a housing 10a in which each circuit in FIG. 10 is stored and a lens barrel 12b in which an optical system 12a is stored. A shutter button 15a forming the operation unit 15 is provided on the upper surface of the housing 10a.

  Further, the recorder 70 shown in FIG. 11 has a housing 70a in which each circuit of FIG. 1 is housed. The housing 70a has a cubic shape, and two end microphones of two microphones 72R and 72L that form an ST sound pickup section 72 are provided so as to project therefrom.

  In FIG. 10, a camera 60 constituting the image pickup apparatus is different from the camera 10 of FIG. 1 in that a recording unit 61 is adopted instead of the recording unit 17 and an image feature extracting unit 14 is added. The image feature extraction unit 14 is configured by a processor and the like, and is configured to extract the image feature of the image captured and acquired by the imaging unit 12 and output the image feature to the control unit 11. For example, the image feature extraction unit 14 can extract information such as the position and size of the main subject on the screen as image features.

  The recording unit 61 records the captured image provided from the control unit 11 and the stereo internal sound collected. The recording section 61 is provided with a cooperation information section 61a. Information related to communication between the recorder 70 and the playback device 80 is recorded in the cooperation information unit 61a, and the control unit 11 controls the communication units 18a and 18b based on the information read from the cooperation information unit 17b. As a result, information can be exchanged between the recorder 70 and the reproduction device 80 by communication. The control unit 11 transmits the audio feature information of the internal sound (first sound) to the recorder 70, and at the same time, can transmit the image and the internal sound obtained by imaging to the playback device 80. Is becoming The control unit 11 can also transmit the information of the image feature extracted by the image feature extraction unit 14 to the recorder 70 at the same time as the information of the audio feature of the internal voice.

  The recorder 70 is different from the external sound collecting device 20 of FIG. 1 in that an ST sound collecting unit 72, a control unit 71, and a recording unit 73 are adopted in place of the multi sound collecting unit 22, the control unit 21, and the recording unit 25, respectively. .. The ST sound collecting unit 72 has two microphones 72R and 72L. The microphones 72R and 72L have, for example, the same characteristics as each other, and are arranged such that the sensitivity peak directions are different from each other by a predetermined angle.

  The control unit 71 includes a sound collection control unit 71a and a sound acquisition unit 71b, and the sound collection control unit 71a controls the sound collection of the ST sound collection unit 72. The sound acquisition unit 71b can acquire the external sound collection sound collected by the microphones 72R and 72L of the ST sound collection unit 72 as stereo sound. The control unit 71 is configured to give an external sound collection sound, which is a stereo sound collected by the ST sound collection unit 72, to the recording unit 73 and record the sound.

  The recording unit 73 is provided with a cooperation information unit 73a. Information related to communication between the camera 60 and the playback device 80 is recorded in the cooperation information unit 73a, and the control unit 71 controls the communication units 26a and 26b based on the information read from the cooperation information unit 73a. Thus, information can be exchanged between the camera 60 and the playback device 80 by communication. The control unit 71 can receive the information of the audio feature of the internal audio (first audio) from the camera 60 and can also transmit the external audio to the playback device 80.

  In the present embodiment, the voice acquisition unit 71b can adjust the two externally collected voices that have been collected based on the voice characteristics of the received internal voice to acquire the external voice. .. For example, the sound acquisition unit 71b may be configured to acquire, as the external sound, stereo sound in which the phases and levels of the two collected external sound are individually adjusted based on the sound feature of the internal sound. ..

  For example, in the camera 60, when the internal sound is picked up in a state where the subject is located in the center of the screen, the phases and levels of the two internal sounds, which are stereo sounds acquired by the ST sound pickup unit 13, are substantially the same. it is conceivable that. On the other hand, it is conceivable that the phase and level of the sound of the subject picked up by the microphones 72R, 72L may be different from each other depending on the relationship between the positions of the subject and the microphones 72R, 72L. Therefore, by adjusting the phase and level of the external sound pickup sound picked up by the microphones 72R and 72L, it is possible to acquire the stereo sound having the matched phase and level as the external sound.

  Further, the sound acquisition unit 71b may change the adjustment amount based on the distance from the ST sound collecting unit 13 to the subject and the distance from the ST sound collecting unit 72 to the subject. FIG. 12 is an explanatory diagram for explaining the difference between the angle from the ST sound collecting unit 13 to the subject and the angle from the ST sound collecting unit 72 to the subject. The example of FIG. 12 shows an example in which the recorder 70 is arranged on the optical axis of the camera 60. As shown in FIG. 12, when the subject is located at a position displaced by X1 from the optical axis of the camera 60, the distance from the optical axis of the camera 60 (ST sound collecting unit 13) to the subject is relatively long. While the angle is θ1, the angle from the optical axis is θ2, which is larger than θ1, for the recorder 70 (ST sound pickup unit 72) whose distance to the subject is relatively short. Therefore, by changing the adjustment amount of the external sound pickup sound according to the distance between the camera 60 and the subject of the recorder 70, it is possible to acquire the external sound more suitable for the image.

  Further, the sound acquisition unit 71b may be configured to individually adjust the phases and levels of the two externally collected sounds by using the image feature information. For example, when the image feature information indicates that the subject is located at the edge of the screen, the phase and level of the external sound collection sound are made to differ from each other based on the image feature, thereby Corresponding to the position, it is possible to make the localization of the sound emitted by the subject for the photographer and the sound image localization obtained by the external sound substantially coincide. Note that the voice acquisition unit 71b may perform adjustment in the frequency domain or the like on the two externally collected voices, not limited to the phase and the level.

  The audio acquisition unit 71b transmits the acquired external audio to the playback device 80. The audio acquisition unit 71b may transmit the stereo sound of which the phase and level are adjusted to the reproduction device 80 as the external sound, and the stereo sound before the adjustment of the phase and the level and the adjustment value are used as the external sound information. It may be transmitted to the playback device 80.

  The playback device 80 may be configured by a computer, a smartphone, a tablet terminal, or the like. The playback device 80 includes a control unit 81. The control unit 81 may be configured by a processor using a CPU, an FPGA, or the like, operate according to a program stored in a memory (not shown) to control each unit, or may be a function of an electronic circuit of hardware. Some or all may be realized.

  The playback device 80 is provided with an operation unit 83. The operation unit 83 includes various switches, dials, ring members and the like (not shown) for reproducing mode setting, parameter operation, etc., and outputs an operation signal based on a user operation to the control unit 81. The control unit 81 controls each unit based on an operation signal from the operation unit 83. The communication unit 82 is controlled by the control unit 81, and can communicate with the camera 60 and the recorder 70 to exchange information. The control unit 81 receives video and internal sound from the camera 60 and external sound from the recorder 70 via the communication unit 82.

  The control unit 81 is provided with a synchronous reproduction unit 81a, and the synchronous reproduction unit 81a can synchronize the received internal audio, external audio, and video. For example, the synchronous playback unit 81a synchronizes the video of the subject with the external audio by comparing the waveform of the audio signal picked up at the same time as the video signal acquired by the imaging unit 12 with the waveform of the external audio. It may be like this.

  The reproducing device 80 is provided with a reproducing unit 84 and a recording unit 85. The recording unit 85 is provided with the internal sound, the external sound, and the image received from the control unit 81, and records them in synchronization with each other. The reproducing unit 84 includes a display unit and a speaker (not shown), and is controlled by the control unit 81 to reproduce and output external audio and video synchronized by the synchronous reproducing unit 81a.

  Next, the operation of the embodiment configured as described above will be described with reference to FIGS. 13 to 15. 13 is a flow chart for explaining the operation of the camera 60, FIG. 14 is a flow chart for explaining the operation of the recorder 70, and FIG. 15 is a flow chart for explaining the operation of the playback device 80.

  Now, it is assumed that the camera 10 and the external sound collecting device 20 are replaced with the camera 60 and the recorder 70 in the above-described examples of FIGS. 3 and 4, respectively. That is, a user who has the recorder 70 disposed at a position relatively close to the trees 31a and 31b and carries the camera 60 is separated from the bird 41 relatively in order to prevent the bird 41 from escaping and to avoid obstacles. Shoot at the desired position.

  Therefore, even in this case, if the sound of the bird 41 is picked up, the ST sound collecting unit of the recorder 70 is more than the sound collecting sound (first sound) by the ST sound collecting unit 13 built in the camera 60. It is considered that a better S / N can be obtained by the external sound pickup voice (second voice) by the microphones 72R, 72L of 72.

  When the power is turned on, the control unit 11 of the camera 60 determines in step S41 of FIG. 13 whether or not the imaging mode is instructed. When the shooting mode is not instructed, the control unit 11 sets the specified mode, for example, the cooperation mode for performing settings and transmission / reception for cooperation with the recorder 70 and the reproduction device 80, or the reproduction for reproducing the recorded image. Switch to mode.

  When the imaging mode is instructed, in the next step S42, the control unit 11 starts shooting a moving image and starts collecting sound by the ST sound collecting unit 13, and recording the moving image and the internal sound, which is stereo sound, in the recording unit 61. To start. When the ending operation is performed, the control unit 11 ends the recording and the sound recording, and converts the video and the internal sound of the recording unit 61 into a file.

  The control unit 11 determines in step S43 whether or not the cooperation with the recorder 70 is designated. When the cooperation is not designated, the control unit 11 returns the process to step S41, and when the cooperation is designated, moves the process to step S44.

  In the next step S44, the feature extraction unit 11g of the control unit 11 extracts the voice feature of the internal voice (first voice) that is the stereo voice from the ST sound pickup unit 13 and removes noise from the extracted voice feature. Then, it transmits to the recorder 70 via the communication parts 18a and 18b (step S45).

  In addition, the image feature extraction unit 14 determines an image corresponding to sound, that is, a main subject from the captured image (step S46), extracts image features and transmits them to the recorder 70 (step S47), and the process proceeds to step S41. return. Note that the processes of steps S46 and S47 can be omitted when the recorder 70 does not use the image feature information for adjusting the externally collected sound.

  On the other hand, when the power is turned on, the control unit 71 of the recorder 70 determines in step S51 of FIG. 14 whether the recording mode is designated. When the recording mode is not designated, the control unit 71 executes another designated mode such as a reproduction mode. When the recording mode is designated, in the next step S52, the control unit 71 starts the sound collection by the ST sound collection unit 72 and starts recording the external sound collection sound, which is a stereo sound, in the recording unit 73.

  The control unit 71 determines in step S53 whether or not the cooperation with the camera 60 is set. When the cooperation is not set, the control unit 71 shifts the processing to step S56 and performs normal recording. That is, the external sound collection sound, which is the stereo sound acquired by the ST sound collection unit 72, is directly recorded in the recording unit 73.

  When the cooperation with the camera 60 is set, the control unit 71 proceeds from step S53 to S54 and acquires the audio feature and the image feature of the internal voice from the camera 60. The control unit 71 records the voice feature of the internal voice in the recording unit 73 together with the external voice pickup using the time information (step S55). As described above, the sound acquisition unit 71b adjusts the phase and level of the external sound collection sound from the ST sound collection unit 72 using the sound feature and the image feature of the internal sound, and obtains the external sound obtained after the adjustment. May be recorded in the recording unit 73.

  In the present embodiment, the video captured by the camera 60 and the external audio captured by the recorder 70 are played back synchronously by the playback device 80. It is assumed that the user operates the operation unit 83 to give an instruction to reproduce a moving image. The control unit 81 determines whether or not the moving image reproduction is designated in step S61 of FIG. 15, and when the moving image reproduction is designated, the process proceeds to step S61. In addition, when the moving image reproduction is not designated, the control unit 81 executes another designated mode.

  In step S <b> 62, the control unit 81 acquires video and internal sound from the camera 60 and external sound from the recorder 70 via the communication unit 82. In addition, the control unit 81 may receive an externally collected sound and an adjustment value as the external sound. In step S63, the synchronous playback unit 81a synchronizes the received internal audio, external audio, and video. That is, the synchronous reproduction unit 81a synchronizes the image of the subject with the external audio by comparing the waveform of the audio signal collected at the same time as the video signal acquired by the imaging unit 12 with the external audio waveform. , And outputs the synchronized video and external audio to the reproduction unit 84.

  The external sound supplied to the reproducing unit 84 is obtained by adjusting the external sound collection sound according to the sound feature and the image feature of the internal sound, and has the sound image localization according to the position of the main subject in the video. In this way, the sound suitable for the video displayed on the display screen of the reproducing unit 84 is output from the speaker of the reproducing unit 84.

  As described above, in the present embodiment, it is possible to output the audio suitable for the video in synchronization regardless of the positional relationship among the imaging device, the external sound collecting device, and the subject.

  In the above embodiment, a digital camera is used as an image capturing device, but the camera may be a digital single lens reflex camera, a compact digital camera, a video camera, a movie camera, or a mobile phone. Of course, a camera built in a personal digital assistant (PDA) such as a smartphone or a smart phone may be used.

  The present invention is not limited to the above embodiments as they are, and can be embodied by modifying the constituent elements within a range not departing from the gist of the invention in an implementation stage. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in each of the above embodiments. For example, some of all the constituent elements shown in the embodiment may be deleted. Furthermore, the constituent elements of different embodiments may be combined appropriately.

  It should be noted that even if the description of the claims, the description, and the operation flow in the drawings is made by using “first,” “next,” and the like for convenience, it is essential that they are performed in this order. It does not mean. Further, it goes without saying that the steps constituting these operation flows can be appropriately omitted as long as they do not affect the essence of the invention.

  Of the techniques described here, the control mainly described in the flowchart is often settable by a program and may be stored in a recording medium or a recording unit. The recording medium and the recording unit may be recorded at the time of product shipment, may be distributed recording medium, or may be downloaded via the Internet.

  It should be noted that, in the embodiments, a portion described as a "section" (section or unit) may be configured by a dedicated circuit or a combination of a plurality of general-purpose circuits, and if necessary, preprogrammed software may be used. It may be configured by combining a microcomputer that operates according to the above, a processor such as a CPU, or a sequencer such as an FPGA. It is also possible to design such that a part or all of the control is taken over by an external device, and in this case, a wired or wireless communication circuit intervenes. The communication may be performed using Bluetooth, WiFi, a telephone line, or the like, and may be performed using USB or the like. A dedicated circuit, a general-purpose circuit, and a control unit may be integrated into an ASIC.

    10 ... Camera, 11 ... Control part, 11a ... Shooting control part, 11b ... Image processing part, 11c ... Focus, angle-of-view information part, 11e ... Sound collection control and processing part, 11f ... Sound acquisition part, 11g ... Feature extraction part , 11h ... Synchronization processing part, 12 ... Imaging part, 12a ... Optical system, 13 ... ST sound collecting part, 14 ... Image feature extracting part, 15 ... Operation part, 16 ... Display part, 17 ... Recording part, 17a ... Sound collection Image-sound recording unit, 17b ... Cooperation information unit, 17c ... External sound recording unit, 17d ... Image-sound DB unit, 18a, 18b, 26a, 26b ... Communication unit, 20 ... External sound collecting device, 21 ... Control unit, 21a ... Sound collection control section, 21b ... Voice acquisition section, 21c ... Feature extraction section, 22 ... Multi sound collection section, 25 ... Recording section, 25a ... Voice information section, 25b ... Linkage information section, 25c ... Voice information section.

Claims (14)

A feature extraction unit that is included in the imaging device and that acquires a first voice that is collected by a built-in sound collection device that collects ambient sound of the imaging device, and that extracts a feature of the acquired first voice;
Based on the feature of the first voice extracted by the feature extractor, at least one of selection and adjustment of a plurality of second voices picked up by a plurality of microphones having different sensitivity distribution directions is performed to obtain a third voice. A voice acquisition unit that
An information acquisition device, comprising: a synchronization processing unit that synchronizes the third sound acquired by the sound acquisition unit with an image of a subject captured by the imaging device. The feature extraction unit acquires the first voice and the second voice, and extracts the feature of the first voice and the feature of the second voice,
The voice acquisition unit performs at least one of selection and adjustment for the plurality of second voices based on a comparison between the feature of the first voice and the feature of the second voice extracted by the feature extraction unit, and a third voice. The information acquisition apparatus according to claim 1, wherein the information acquisition apparatus acquires. The said audio | voice acquisition part acquires said 3rd audio | voice from said some 2nd audio | voice by selection based on the similarity of the characteristic of said 1st audio | voice and the characteristic of 2nd audio | voice. Information acquisition device. The voice acquisition unit acquires, as the third voice, a voice with the highest S / N among a plurality of second voices selected based on the similarity between the feature of the first voice and the feature of the second voice. The information acquisition device according to claim 3, wherein Further equipped with a database of environmental sound characteristics,
The information acquisition apparatus according to claim 2, wherein the feature extraction unit extracts a feature of the first voice using a feature of the environmental sound. The image pickup device further comprises an image feature extraction unit for extracting the features of the image of the subject obtained by imaging,
6. The information acquisition according to claim 5, wherein the feature extraction unit extracts the feature of the first voice by referring to the database based on the feature of the image extracted by the image feature extraction unit. apparatus. The voice acquisition unit adjusts at least one of a phase and a level of the plurality of second voices based on the feature of the first voice extracted by the feature extraction unit to obtain the third voice. The information acquisition device according to claim 1. The information acquisition apparatus according to claim 1, further comprising a recording unit that records the video and the third audio synchronized by the synchronization processing unit. The information acquisition device according to claim 1, further comprising a communication unit that transmits the characteristics of the first sound to the sound acquisition unit when the image is captured by the imaging device. The information acquisition device according to claim 1, wherein the feature extraction unit and the voice acquisition unit are built in at least one of an external sound pickup device having the plurality of microphones and the imaging device. The synchronization processing unit is built in the playback device,
The information acquisition apparatus according to claim 10, wherein the reproduction apparatus includes a communication unit that acquires the video and the third audio from the imaging apparatus and the external sound collection apparatus. An information acquisition method in an information acquisition device including a feature extraction unit, a voice acquisition unit, and a synchronization processing unit,
The feature extracting unit obtains the first voice picked up by a built-in sound pickup device which is built in the image pickup device and picks up ambient sound of the image pickup device, and extracts the feature of the obtained first voice. ,
The voice acquisition unit performs at least one of selection and adjustment for a plurality of second voices picked up by a plurality of microphones having different sensitivity distribution directions based on the extracted feature of the first voice, and a third voice To get
The information acquisition method, wherein the synchronization processing unit synchronizes the acquired third sound with an image of a subject captured by the imaging device. On the computer,
Acquiring the first sound collected by the built-in sound collecting device that is built in the image pickup device and picks up ambient sound of the image pickup device, and extracts the characteristic of the acquired first sound,
Based on the extracted characteristics of the first voice, at least one of selection and adjustment for a plurality of second voices picked up by a plurality of microphones having different sensitivity distribution directions is performed to obtain a third voice,
An information acquisition program for executing a procedure of synchronizing the acquired third sound with the image of the subject obtained by the imaging device. An image pickup apparatus comprising: a built-in sound pickup device that picks up ambient sound; and a feature extraction unit that extracts a feature of the first sound picked up by the built-in sound pickup device,
A plurality of microphones having different sensitivity distribution directions are provided, and a plurality of second sounds picked up by the plurality of microphones are selected and adjusted based on the characteristics of the first sound extracted by the characteristic extraction unit of the imaging device. An external sound pickup device that performs at least one of the above to obtain the third sound,
An information acquisition system, comprising: a reproduction device that performs a synchronization process for synchronizing the third sound acquired by the external sound collecting device with the image of the subject obtained by the image capturing device.

Images