JP2006304257A - Image pickup device, image pickup method, output device, and output method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image pickup device or an output device capable of easily obtaining music being played when an image is picked up as well as the image. <P>SOLUTION: The image pickup device includes: an image pickup section; a recording section for recording sound around the image pickup section; a characteristic sound extraction section for extracting predetermined type of sound from the sound recorded by the recording section; a sound acquisition section for acquiring the same type of sound as the sound extracted by the characteristic sound extraction section from a sound database storing a plurality of types of sound; and a data storage section for storing the sound acquired by the sound acquisition section and an image picked up by the image pickup section while they are associated with each other to be outputted in synchronization. The characteristic sound extraction section may extract predetermined sound from the sound recorded by the recording section within preset time from time when the image pickup section has picked up the image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an imaging device, an imaging method, an output device, and a program. In particular, the present invention relates to an imaging device and an imaging method for capturing an image, an output device and an output method for outputting an image, and a program for the imaging device and the output device.

2. Description of the Related Art Conventionally, there are digital still cameras that can record not only still images but also moving images on a memory card. In such a digital still camera, sound detected by a microphone at the time of shooting and recording of a still image or a moving image can be recorded in association with the image (see, for example, Patent Document 1). There is also known an electronic photo stand that can reproduce sound recorded in correspondence with an image while displaying still images and moving images taken with a digital still camera.
JP-A-7-154734

  With such a camera and an electronic photo stand, for example, a photographed image can be viewed together with music that has been flowing around during photographing recording. However, such a camera can only record the sound that was flowing at the time of shooting and recording. In addition, music recorded from a microphone often has lower sound quality than music data that can be acquired as digital data from, for example, the Internet or a CD. For this reason, the user may be dissatisfied with the music to be played. In addition, it is desirable for the user to be able to easily appreciate the captured image and music without performing complicated operations such as editing of the image and sound. In addition to music, it is desirable to be able to appreciate the same kind of sound as the environmental sound at the time of imaging, such as the sound of waves and the sound of birds, together with images with higher sound quality.

  Accordingly, an object of the present invention is to provide an imaging device, an imaging method, an output device, an output method, and a program that can solve the above-described problems. This object is achieved by a combination of features described in the independent claims. The dependent claims define further advantageous specific examples of the present invention.

  An imaging apparatus according to a first aspect of the present invention includes an imaging unit, a recording unit that records audio around the imaging unit, and a feature sound extraction unit that extracts a predetermined type of audio from the audio recorded by the recording unit And an audio acquisition unit that acquires the same type of audio as the audio extracted by the feature sound extraction unit from an audio database that stores a plurality of types of audio, and the audio acquired by the audio acquisition unit and the image captured by the imaging unit And a data storage unit that stores the data in association with each other so as to output them synchronously.

  The feature sound extraction unit may extract a predetermined type of sound from the sound recorded by the recording unit within a preset time from the time when the image capturing unit captures the image.

  A display unit that displays an image of light received by the light receiving element of the imaging unit and an imaging mode that is an operation mode in which the display unit displays an image, or an operation in which the display unit does not display an image A mode setting unit that sets the imaging apparatus to the non-imaging mode that is the mode, and the recording unit sets the imaging mode when the mode setting unit is set to the imaging mode, and the mode setting unit sets the imaging mode to the non-imaging mode. In both cases, sound around the imaging unit may be recorded.

  The feature sound extraction unit includes the time set by the mode setting unit in the imaging mode, and is recorded by the recording unit within a preset time longer than the time set by the mode setting unit in the imaging mode. A predetermined type of sound may be extracted from The voice database stores a plurality of music, the feature sound extraction unit extracts music from the voice recorded by the recording unit, and the voice acquisition unit is the same as the music extracted by the feature sound extraction unit from the voice database. May get music.

  The feature sound extraction unit further includes a condition storage unit that preliminarily stores a condition for specifying each type of environmental sound extracted by the feature sound extraction unit, and the voice database stores a plurality of environmental sounds for each type of environmental sound. The unit extracts the environmental sound that matches the conditions stored in the condition storage unit from the voice recorded by the recording unit, and the voice acquisition unit extracts the same type of environmental sound from the voice database as the environmental sound extracted by the feature sound extraction unit. The environmental sound may be acquired, and the data storage unit may store the environmental sound acquired by the sound acquisition unit and the image captured by the imaging unit in association with each other so as to be output in synchronization.

  The voice database stores a plurality of music by era, the feature sound extraction unit extracts music from the voice recorded by the recording unit, and the voice acquisition unit extracts the music extracted by the feature sound extraction unit from the voice database. May get music of the same era. The audio database stores a plurality of music by genre, and the audio acquisition unit may acquire music of the same genre as the music extracted by the feature sound extraction unit from the audio database.

  The imaging method according to the second aspect of the present invention includes an imaging stage in which an image is captured by an imaging unit, a recording stage in which sound around the imaging section is recorded, and a predetermined type of sound recorded in the recording stage. Acquired in a feature sound extraction stage for extracting sound, a sound acquisition stage for acquiring the same type of sound as the sound extracted in the feature sound extraction stage, and a sound acquisition stage from a sound database storing a plurality of kinds of sounds A data storage stage for storing the audio and the image captured by the imaging unit in association with each other so as to be output in synchronization.

  According to a third aspect of the present invention, there is provided a program for an imaging device that captures an image, the imaging device including an imaging unit that captures an image, a recording unit that records sound around the imaging unit, and a recording unit A feature sound extraction unit for extracting a predetermined type of sound from the obtained sound, and a sound acquisition unit for acquiring the same type of sound as the sound extracted by the feature sound extraction unit from a sound database storing a plurality of types of sound The voice acquired by the voice acquisition unit and the image captured by the imaging unit are made to function as a data storage unit that stores them in association with each other so as to be output in synchronization.

  An output device according to a fourth aspect of the present invention includes an image storage unit that stores an image captured by an imaging device, an audio storage unit that stores audio recorded by the imaging device, and audio stored in the audio storage unit. A feature sound extraction unit that extracts a predetermined type of sound, and a sound acquisition unit that acquires the same type of sound as the sound extracted by the feature sound extraction unit from a sound database that stores a plurality of types of sound; And an output unit that outputs the sound acquired by the sound acquisition unit and the image stored in the image storage unit in synchronization.

  The image storage unit stores the imaging time of the image in association with the image, the audio storage unit stores the recording time of the audio in association with the audio, and the feature sound extraction unit stores the image A predetermined type of sound may be extracted from sound recorded within a preset allowable time from the time when the image was taken.

  When the difference between the output request acquisition unit that acquires the output request of the image stored in the image storage unit, the time when the output request acquisition unit acquires the output request, and the imaging time of the image stored in the image storage unit is larger And an allowable time setting unit for setting the allowable time longer.

  An output method according to a fifth aspect of the present invention includes an image storage stage for storing an image captured by an imaging apparatus, an audio storage stage for storing sound recorded by the imaging apparatus, and an audio stored in the audio storage stage. A feature sound extraction stage for extracting a predetermined type of voice from the voice acquisition stage for acquiring the same type of voice as the voice extracted in the feature sound extraction stage from a voice database storing a plurality of music; An output stage for synchronizing and outputting the audio acquired in the audio acquisition stage and the image stored in the image storage stage.

  According to a sixth aspect of the present invention, there is provided a program for an output device that outputs an image, wherein the output device is recorded by an imaging unit that captures an image, a recording unit that records sound around the imaging unit, and a recording unit A feature sound extraction unit for extracting a predetermined type of sound from the obtained sound, and a sound acquisition unit for acquiring the same type of sound as the sound extracted by the feature sound extraction unit from a sound database storing a plurality of types of sound The voice acquired by the voice acquisition unit and the image captured by the imaging unit are made to function as a data storage unit that stores them in association with each other so as to be output in synchronization.

  Note that the above summary of the invention does not enumerate all the necessary features of the present invention, and sub-combinations of these feature groups can also be the invention.

  ADVANTAGE OF THE INVENTION According to this invention, the imaging device and output device which can obtain easily the kind of audio | voice which was flowing when the image was imaged with an image can be provided.

  Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the claimed invention, and all combinations of features described in the embodiments are inventions. It is not always essential to the solution.

  FIG. 1 shows an example of a voice providing system according to an embodiment of the present invention. The voice providing system includes an imaging device 100, an output device 140, an explanatory sound database 170, a music database 172, and an environmental sound database 174. In this example, the imaging device 100 captures a commemorative photo at a sightseeing spot. The image capturing apparatus 100 transmits the image captured by the image capturing apparatus 100 and the image capturing position when the image capturing is performed to the output apparatus 140 through the communication line 150 such as the Internet. The music database 172 stores a plurality of music. The environmental sound database 174 stores a plurality of environmental sounds for each type of environmental sound. Note that the music database 172 and the environmental sound database 174 in this embodiment are examples of the audio database in the present invention.

  The output device 140 acquires audio data relating to the imaging position received from the imaging device 100, for example, audio data describing the feature of the tourist spot from the explanatory sound database 170, and outputs it together with the image received from the imaging device 100. Note that the imaging apparatus 100 may record voices around the imaging apparatus 100, and the output apparatus 140 may receive voices recorded by the imaging apparatus 100. The output device 140 may extract music from the sound, acquire the same music as the extracted music from the music database 172, and output the music together with the image. Further, the output device 140 extracts environmental sounds such as wave sounds and bird calls from the sound, acquires environmental sounds of the same type as the extracted environmental sounds from the environmental sound database 174, and together with the images. It may be output. Note that the output device 140 may simultaneously output the sound data acquired from the explanatory sound database 170 and the music acquired from the music database 172 or the environmental sound acquired from the environmental sound database 174.

  The output device 140 may be a device that outputs images and sounds, such as an HDTV, an electronic photo stand, and a computer. Further, the output device 140 may output the voice as characters. For example, when the output device 140 displays an image on a display device such as a liquid crystal display, the voice acquired from the explanatory sound database 170 and / or the lyrics included in the music data acquired from the music database 172 are displayed on the display device as characters. You may display. Note that the output device 140 may display characters on a display device that displays an image, and may display characters on a display device that is different from the display device that displays an image. In addition, the output device 140 may be a printing device that prints an image, such as a printer, and may print the image and sound as characters.

  The imaging device 100 may be, for example, a digital still camera, a camera-equipped mobile phone, or the like. Further, the image capturing apparatus 100 may record image and sound data on a recording medium, and the output apparatus 140 may receive the image and sound data from the recording medium. In addition, the imaging apparatus 100 may store the image and audio data in a directory provided for each user 180 of the server connected to the communication line 150, for example, a directory associated with the imaging apparatus 100. The output device 140 may receive image and sound data stored in the server for each user 180.

  According to the output device 140 described above, the image captured by the imaging device 100 can be provided to the user 180 together with the sound related to the captured location. For this reason, the user 180 can enjoy the image happily while remembering features such as sightseeing spots. In addition, music that has been flowing around when imaged by the imaging apparatus 100, environmental sounds such as ambient sound, and the like can be provided to the user 180. For this reason, the user 180 can enjoy the image happily while listening to the music that was popular when visiting tourist spots.

  FIG. 2 shows an example of a block configuration of the output device 140. The output device 140 includes an image storage unit 210, an image selection unit 278, an imaging region determination unit 282, an imaging period determination unit 284, an imaging position distribution calculation unit 286, an imaging number calculation unit 288, an output unit 224, and an audio acquisition unit 262. Prepare.

  The image storage unit 210 stores the captured image and the position where the image is captured in association with each other. For example, the imaging device 100 receives latitude and longitude information at the position of the imaging device 100 when an image is captured from a GPS satellite. The image storage unit 210 stores the image received from the imaging device 100 in association with the latitude and longitude information detected by the imaging device 100.

  The image selection unit 278 selects a plurality of images from the images stored in the image storage unit 210 based on an instruction input from the user 180. The imaging region determination unit 282 determines an imaging region including a plurality of imaging positions where each of the plurality of images stored in the image storage unit 210 is captured.

  Specifically, the imaging region determination unit 282 determines an imaging region including a plurality of positions where each of the plurality of images selected by the image selection unit 278 is captured. For example, the imaging region determination unit 282 may determine a geographical range of an imaging region including a plurality of imaging positions where each of the plurality of images stored in the image storage unit 210 is captured, for example, latitude and longitude ranges. .

  The imaging position distribution calculation unit 286 calculates a distribution of a plurality of positions at which each of a plurality of images stored in the image storage unit 210 in the imaging region determined by the imaging region determination unit 282 is captured. The number of captured images calculation unit 288 calculates the number of captured images in which each of the plurality of images stored in the image storage unit 210 is captured for each of the plurality of partial regions included in the imaging region determined by the imaging region determination unit 282.

  The voice acquisition unit 262 acquires the voice stored in the explanatory sound database 170 according to the size of the imaging area determined by the imaging area determination unit 282. Specifically, when the imaging area determined by the imaging area determination unit 282 is narrower, the audio acquisition unit 262 acquires a more detailed description of the audio regarding the imaging area determined by the imaging area determination unit 282.

  More specifically, the sound acquisition unit 262 acquires a description sound for a partial region in which the distribution bias calculated by the imaging position distribution calculation unit 286 is larger than a predetermined bias. In addition, when the number of captured images calculated by the captured image number calculating unit 288 is larger, the sound acquisition unit 262 has a more detailed description of a plurality of descriptions with different details about partial areas stored in the description sound database 170. To get.

  The output unit 224 outputs the sound acquired by the sound acquisition unit 262 together with the plurality of images stored in the image storage unit 210. Specifically, the output unit 224 outputs the sound acquired by the sound acquisition unit 262 together with the plurality of images selected by the image selection unit 278.

  The image storage unit 210 further stores the time when the image was captured in association with the captured image. The imaging period determination unit 284 determines an imaging period including a plurality of times when each of the plurality of images stored in the image storage unit 210 is captured. Specifically, the imaging period determination unit 284 determines an imaging period including a plurality of times when a plurality of images selected by the image selection unit 278 are captured.

  Then, the voice acquisition unit 262 acquires the voice stored in the explanatory sound database 170 according to the length of the imaging period determined by the imaging period determination unit 284. Specifically, when the imaging period determined by the imaging period determination unit 284 is longer, the voice acquisition unit 262 acquires a more detailed explanation voice regarding the imaging region determined by the imaging region determination unit 282.

  FIG. 3 shows an example of data stored in the explanatory sound database 170 in a table format. The explanatory sound database 170 stores a position range indicating a range where the region is located and sound data regarding the region in association with the region. The position range may include, for example, the latitude and longitude that are the starting point of the region including the region, and the latitude and longitude that are the end point. Note that the position range may include a plurality of ranges where the region is located. The audio data may include, for example, a plurality of audio data with different details about each region, for example, audio data with an outline description and a detailed description. Further, the audio data may include news about each area.

  For example, the outline explanation data about Japan may be voice data explaining characteristics, population, area, etc. of the whole of Japan. Moreover, the audio data explaining the details about Japan may be audio data explaining the characteristics, population, area, etc. of each region included in Japan, or even audio data explaining the history of Japan. Good.

  FIG. 4 shows an example of the distribution of positions at which images are taken. In the example of FIG. 4, the image storage unit 210 captures 10 images, 9 images, 10 images, 7 images, 8 images, and 6 images captured in Nagasaki, Hiroshima, Kyoto, Shizuoka, Tokyo, and Hokkaido, respectively. Storing. The imaging area determination unit 282 determines, for example, a partial area including a position where an image is captured by dividing into partial areas for each predetermined latitude and longitude range. The number of captured images calculation unit 288 calculates the number of images captured in each of the partial areas. Then, the imaging position distribution calculating unit 286 determines the distribution of the partial area including the position where the image is captured.

  Then, the sound acquisition unit 262 determines whether the imaging positions are distributed over a wide range. For example, when the image picked up in Nagasaki is selected by the image selection unit 278, the sound acquisition unit 262 determines that the image pickup positions are distributed in a narrow range, and as sound data explaining Japan in more detail, for example, Audio data describing Nagasaki is acquired from the explanatory sound database 170. When images picked up in Nagasaki, Hiroshima, Kyoto, Shizuoka, Tokyo, and Hokkaido are selected by the image selection unit 278, it is determined that the image pickup positions are distributed over a wider range, For example, voice data explaining Japan is acquired from the explanation sound database 170 as voice data explaining the outline of the above.

  Note that when the number of images calculated by the imaging number calculation unit 288 is larger, the voice acquisition unit 262 acquires more detailed explanation audio data for each imaging region from the explanation sound database 170, and the number of images is smaller. In such a case, audio data that explains the outline of each imaging region may be acquired from the explanatory sound database 170.

  In addition, the sound acquisition unit 262 calculates the ratio of the number of images captured in the specific partial area to the selected number, and when the ratio is larger than a predetermined ratio, the specific partial area Is obtained from the explanation sound database 170. For example, the image storage unit 210 stores a total of 50 images of 6, 7, 30, 4, and 3 images captured in Nagasaki, Hiroshima, Kyoto, Tokyo, and Hokkaido, respectively. When the above 30 images are images captured in Kyoto, the sound acquisition unit 262 acquires sound data describing Kyoto from the description sound database 170. For this reason, the output device 140 can provide the user 180 with sound that describes a place where the user 180 has captured a particularly large amount.

  FIG. 5 shows an example of the distribution of times when images are captured. The imaging period determination unit 284 determines a time range in which an image is captured for each partial area. For example, the imaging period determination unit 284 includes time ranges (t1 to t10, t11 to t19, t20 to t29, t30 to t36, which are captured in partial areas including each of Nagasaki, Hiroshima, Kyoto, Shizuoka, Tokyo, and Hokkaido. t37 to t44 and t45 to t50) are determined.

  Then, the audio acquisition unit 262 determines the length of the imaging period. For example, when the image picked up in Nagasaki is selected by the image selection unit 278, the sound acquisition unit 262 determines that the period (t1 to t10) picked up in Nagasaki is shorter, and obtains sound data describing Nagasaki. Obtained from the explanatory sound database 170. When images picked up in Nagasaki, Hiroshima, Kyoto, Shizuoka, Tokyo, and Hokkaido are selected by the image selection unit 278, it is determined that the imaged period (t1 to t50) is longer, and the imaging region For example, Japanese detailed audio data is acquired from the explanatory sound database 170.

  Then, the sound acquisition unit 262 calculates the ratio of the period when the selected image is captured with respect to the period when the selected image is captured, and if the ratio is greater than a predetermined ratio, The voice data describing the partial area is acquired from the explanation sound database 170. For example, when the image storage unit 210 stores images captured in Nagasaki, Hiroshima, Kyoto, Tokyo, and Hokkaido, the selected image is captured during the period (t64 to t93) captured in Kyoto. Audio acquisition unit 262 receives the audio data for explaining the Kyoto data when the period is equal to or longer than half of the total period (total period of t51 to t56, t57 to t63, t64 to t93, t94 to t97, t98 to t10). Obtained from the explanatory sound database 170. For this reason, the output device 140 can provide the user 180 with a sound that explains the location where the user 180 stayed for a long time and captured the image.

  FIG. 6 shows an example of a block configuration of the imaging apparatus 600. The imaging apparatus 600 is another example of the imaging apparatus 100 described with reference to FIG. 1, and has a function of extracting and storing characteristic sounds such as music and environmental sounds from voices recorded together with captured images. Have. The imaging apparatus 600 includes a mode setting unit 692, an imaging unit 602, a display unit 670, a data storage unit 698, a recording unit 650, a characteristic sound extraction unit 694, a condition storage unit 660, and a voice acquisition unit 696.

  The imaging unit 602 captures an image. Specifically, the imaging unit 602 captures the subject by receiving light from the subject with an imaging device such as a CCD. Note that the imaging unit 602 may continuously capture the subject at predetermined time intervals. Then, the imaging unit 602 may hold a predetermined number of images obtained by continuously capturing images. Then, the imaging unit 602 may use, as the captured image, an image captured at the timing closest to the time when the imaging is instructed, among the images held by the imaging unit 602.

  The display unit 670 displays an image of light received by the light receiving element included in the imaging unit 602. The mode setting unit 692 includes the imaging device 600 in an imaging mode that is an operation mode in a state where the display unit 670 is displaying an image or a non-imaging mode that is an operation mode in which the display unit 670 is not displaying an image. Set.

  The recording unit 650 records the sound around the imaging unit 602. Note that the recording unit 650 records the sound around the imaging unit 602 both when the mode setting unit 692 is set to the imaging mode and when the mode setting unit 692 is set to the non-imaging mode. .

  The feature sound extraction unit 694 extracts a predetermined type of sound from the sound recorded by the recording unit 650. For example, the feature sound extraction unit 694 extracts music from the sound recorded by the recording unit 650. In this case, the feature sound extraction unit 694 extracts the fundamental frequency based on the frequency spectrum of the speech and determines the scale. Then, the feature sound extraction unit 694 determines music feature values such as rhythm, tempo, and tonality based on the determined scale, and extracts note data. The feature sound extraction unit 694 may further extract the chord progression of music based on the note data.

  The condition storage unit 660 stores in advance conditions for specifying each type of environmental sound extracted by the feature sound extraction unit 694. Specifically, the condition storage unit 660 stores a frequency spectrum that is characteristic of each type of environmental sound for each type of environmental sound, such as a dog, bird call, insect call, and wave sound. Then, the feature sound extraction unit 694 extracts the environmental sound that matches the conditions stored in the condition storage unit 660 from the sound recorded by the recording unit 650. For example, the feature sound extraction unit 694 extracts the environmental sound that matches the frequency spectrum stored in the condition storage unit 660 with a predetermined matching degree or higher from the sound recorded by the recording unit 650. The condition storage unit 660 may store the environmental sound itself for each type of environmental sound. In this case, the feature sound extraction unit 694 compares the sound of the environmental sound stored in the condition storage unit 660 and the sound recorded by the recording unit 650, and the feature amount (for example, frequency spectrum) of the sound is the most coincident. A high environmental sound may be extracted to determine the type of the environmental sound.

  The feature sound extraction unit 694 extracts a predetermined type of sound from the sound recorded by the recording unit 650 within a preset time from the time when the image capturing unit 602 captured the image. For example, the feature sound extraction unit 694 extracts music or environmental sound from the sound recorded by the recording unit 650 within a preset time from the time when the image capturing unit 602 captured the image. Specifically, the feature sound extraction unit 694 includes the time set by the mode setting unit 692 in the imaging mode and within a preset time longer than the time set by the mode setting unit 692 in the imaging mode. A predetermined type of sound is extracted from the sound recorded by the recording unit 650. More specifically, the feature sound extraction unit 694 includes a time set in advance by the mode setting unit 692 including the time set in the imaging mode and longer than the time set by the mode setting unit 692 in the imaging mode. The music or environmental sound is extracted from the voice recorded by the recording unit 650.

  The music database 172 stores a plurality of music by era. The music database 172 stores a plurality of music by genre. Specifically, the music database 172 stores music genres and times in association with music data. The music database 172 may store music feature values such as note data, rhythm, tempo, tonality, and chord progression in association with the music data. In addition, the music database 172 may store a person related to the music, for example, a composer, a lyricist, an arranger, a performer, or the like in association with the music data. Further, the music database 172 may store a transmission position indicating a position of a region where the music is transmitted, a transmission time at which the music is transmitted, and a transmission unit in association with the music data. Note that the transmission means may be, for example, a radio or a cable. The transmission time may be information indicating the time at which music is to be broadcast, such as program data of a broadcasting station. The music database 172 may store information indicating the degree to which the music hits in a plurality of regions in association with the music data for each region and each era.

  The voice acquisition unit 696 acquires the same type of voice as the voice extracted by the feature sound extraction unit 694 from a voice database that stores a plurality of types of voice. Specifically, the voice acquisition unit 696 acquires the same music as the music extracted by the feature sound extraction unit 694 from the music database 172. Specifically, the voice acquisition unit 696 acquires from the music database 172 music having note data that matches the note data extracted by the feature sound extraction unit 694. At this time, the sound acquisition unit 696 detects the imaging position and imaging time of the imaging device 600 at the timing when the imaging unit 602 captures an image, and associates the transmission position including the imaging position and the transmission time including the imaging time. The same music as the music extracted by the feature sound extraction unit 694 may be acquired from the music data stored in the music database 172. At this time, the voice acquisition unit 696 preferentially searches for and acquires music having a higher degree of hit in the era including the region including the imaging position and the imaging time among the music data stored in the music database 172. Good. In addition, the voice acquisition unit 696 may specify a transmission unit to which music is to be transmitted based on the imaging position and sequentially search from the music transmitted by the transmission unit. For example, when acquiring music recorded in a residential area, the audio acquisition unit 696 searches in order from music to be broadcast on radio broadcast, and when acquiring music recorded at an amusement park, You may search in order from the music which should be broadcast by cable broadcasting.

  In addition, the voice acquisition unit 696 acquires the same type of music as the music extracted by the feature sound extraction unit 694 from the music database 172 storing a plurality of music. Specifically, the voice acquisition unit 696 acquires, from the music database 172, music of the same era as the music extracted by the feature sound extraction unit 694. Also, the voice acquisition unit 696 acquires music of the same genre as the music extracted by the feature sound extraction unit 694 from the music database 172. Specifically, the voice acquisition unit 696 specifies and specifies the genre and / or period of music based on feature quantities such as rhythm, tempo, tonality, and chord progression extracted by the feature sound extraction unit 694. Genre and / or period music is obtained from the music database 172. In addition, the voice acquisition unit 696 specifies a person related to the music by searching the music database 172 for music of the feature amount based on the feature amount extracted by the feature sound extraction unit 694. Music associated with the specified person may be acquired from the music database 172. Note that the voice acquisition unit 696 may acquire music having the highest degree of hit among the music data stored in the music database 172 in the region including the imaging position and the period including the imaging time.

  In addition, the sound acquisition unit 696 acquires from the environmental sound database 174 the same type of environmental sound as the environmental sound extracted by the feature sound extraction unit 694. When the condition storage unit 660 stores the environmental sound itself, the sound acquisition unit 696 may acquire the same type of sound as the sound extracted by the feature sound extraction unit 694 from the condition storage unit 660.

  The data storage unit 698 stores the audio acquired by the audio acquisition unit 696 and the image captured by the imaging unit 602 in association with each other so as to be output in synchronization. Specifically, the data storage unit 698 stores the music acquired by the voice acquisition unit 696 and the image captured by the imaging unit 602 in association with each other so as to be output in synchronization. In addition, the data storage unit 698 stores the environmental sound acquired by the audio acquisition unit 696 and the image captured by the imaging unit 602 in association with each other so as to be output in synchronization. According to the imaging device 600 described above, it is possible to provide the user 180 with the same music as the BGM flowing around the imaging device 600 when the imaging device 600 takes an image. In addition, the imaging apparatus 600 can provide ambient sound at the time of imaging to the user 180 together with the image.

  FIG. 7 shows an example of data stored in the music database 172 in a table format. The music database 172 is associated with music data, the era, the genre to which the music belongs, the rhythm that the music has, the tempo that the music has, the tonality that the music has, the chord progression that the music has, the note data that indicates the musical score, the music Stores the record company that owns the original title, the label that produced the music, and the hit degree of the music hit. Note that the times when the music database 172 stores may be the times when music was created, the times when music was released, the times when music was popular, and the like. Note that the music database 172 includes various attributes related to music such as a producer that produced music, a music album that includes music, and billing information that is charged when music is downloaded, in addition to the attributes illustrated in FIG. Needless to say, can be stored.

  FIG. 8 shows an example of sound and time range recorded by the imaging apparatus 600. The imaging apparatus 600 has an imaging mode, an output mode, and a standby mode as operation modes. The imaging mode may be an operation mode in which the imaging apparatus 600 can capture and / or record. The output mode may be an operation mode in which the imaging apparatus 600 can output an image and / or sound, for example. Note that the non-imaging mode includes an output mode and a standby mode. The imaging apparatus 600 includes a period (t1 to t3) set in the imaging mode, a period (t3 to t4) set in the output mode, and a period (t0 to t1 and t4) set in the standby mode. Sounds around the imaging apparatus 600 at t5) are recorded.

  Note that immediately after the imaging apparatus 600 is activated, the imaging apparatus 600 is set to the standby mode. When the operation mode is set to the standby mode or the output mode, the imaging apparatus 600 transitions to the imaging mode when an operation related to the imaging operation or the recording operation is performed by the user 180. The operation related to the imaging operation includes, for example, an operation for imaging an image, an operation for adjusting imaging conditions such as a shutter speed and a focal length, and the like. The operation related to the recording operation includes, for example, an operation for adjusting recording conditions such as an operation for recording sound and an adjustment of recording sensitivity. In addition, when the operation mode is set to the standby mode or the imaging mode, the imaging apparatus 600 transitions to the output mode when an operation related to the output operation of the imaging apparatus 600 is performed by the user 180. The operation related to the output operation includes, for example, an operation for outputting an image, an operation for selecting an image to be output, an operation for adjusting output conditions such as adjustment of an output speed, and the like. Note that when the imaging apparatus 600 is set to the imaging mode or the output mode, the imaging apparatus 600 transitions to the standby mode on condition that the operation of the imaging apparatus 600 by the user 180 is not operated for a predetermined period. It's okay.

  The feature sound extraction unit 694 extracts music from the sound recorded by the recording unit 650 in a time range before or after a predetermined time from the imaging time t2. For example, when the imaging apparatus 600 is instructed by the user 180 to capture an image at time t2, the feature sound extraction unit 694 includes a standby period including a period (t1 to t3) set in the imaging mode including the imaging time t2. Music is extracted from the sound recorded during the period set to the mode or the output mode, for example, the period (t0 to t5).

  Note that the feature sound extraction unit 694 extracts music from the sound recorded at the time closest to the imaging time t2 among the sounds recorded by the recording unit 650 in the period (t0 to t5) including the imaging time t2. It's okay. Further, the feature sound extraction unit 694 may extract music from the sound with the loudest volume.

  FIG. 9 shows an example of a procedure for acquiring music. The feature sound extraction unit 694 extracts the sound in the period including the time taken from the sound recorded by the recording unit 650 (S912). Then, the feature sound extraction unit 694 extracts a feature amount of music from the sound of the period extracted in S912 (S914). The feature amount of music may be, for example, note data, rhythm, tempo, tonality, chord progression, or the like.

  Then, the voice acquisition unit 696 searches the music database 172 for music that matches the note data extracted by the feature sound extraction unit 694 (S916). Then, the feature sound extraction unit 694 determines whether or not music that matches the extracted note data is stored in the music database 172 (S918). If it is determined in S918 that there is matching music, the voice acquisition unit 696 acquires matching music from the music database 172 (S920).

  If the sound acquisition unit 696 determines that there is no matching music in S918, based on the feature amount of the music extracted in S914 by the feature sound extraction unit 694, the same genre and / or age as the music extracted in S912 Is specified (S922). For example, the voice acquisition unit 696 identifies a genre and / or era that includes the most music having the most similar feature amount among the music stored in the music database 172. Then, the voice acquisition unit 696 acquires the music of the genre and / or age specified in S922 from the music stored in the music database 172 (S924). In S922, the voice acquisition unit 696 may specify the record company or label stored in the music database 172 in association with the most music having similar feature values in addition to the genre and age. . In S924, the voice acquisition unit 696 may acquire the music of the record company or label specified in S922 from the music stored in the music database 172. In S924, the voice acquisition unit 696 may acquire the music with the highest hit degree stored in the music database 172 when there are a plurality of the same type of music.

  When a plurality of music determined to be the same music in S918 are searched from the music database 172, the voice acquisition unit 696 acquires the plurality of searched music from the music database 172 and causes the user 180 to select the music. Also good. Also, in S924, the voice acquisition unit 696 may acquire a plurality of pieces of music of the same genre and / or age acquired from the music database 172 and allow the user 180 to select them. In addition, the voice acquisition unit 696 may acquire music having the highest hit degree stored in the music database 172 among the plurality of searched music. In addition, when a plurality of music is searched, the voice acquisition unit 696 stores music of a record company or label stored in the music database 172 in association with the largest number of music among the plurality of searched music. May be specified. Then, the voice acquisition unit 696 may acquire the music of the specified record company or label from the music stored in the music database 172. In S920, the voice acquisition unit 696 may acquire music from the same record company or label as the music determined to be the same music in S918 from the music database 172. In addition to the record company or the label, the voice acquisition unit 696 acquires, from the music database 172, music stored in the music database 172 in association with the same attribute as the music determined to be the same music in S918. May be.

  As described above, the operation in which the sound acquisition unit 696 acquires music from the music database 172 has been described with reference to FIGS. 7 to 9. However, the sound acquisition unit 696 can acquire the environmental sound from the environmental sound database 174 by the same operation. it can. Needless to say, the sound acquisition unit 696 may acquire various types of predetermined sounds to be recorded in association with images, in addition to music and environmental sounds, from the sound database.

  FIG. 10 shows an example of a block configuration of the output device 1040. The output device 1040 may be another example of the output device 140 included in the voice providing system. In addition to the captured image and the surrounding sound at the time of capturing, the image capturing apparatus 100 also transmits the time when the surrounding sound is recorded to the output device 1040.

  The output device 1040 includes an image storage unit 1010, an output unit 1024, an audio acquisition unit 1096, a feature sound extraction unit 1094, a condition storage unit 1060, an audio storage unit 1020, an allowable time setting unit 1043, an output time detection unit 1044, and an output request acquisition. Part 1048.

  The image storage unit 1010 stores an image captured by the imaging device 100. The image storage unit 1010 stores the imaging time when the image is captured in association with the image.

  The sound storage unit 1020 stores the sound recorded by the imaging device 100. The voice storage unit 1020 stores the recording time of the voice in association with the voice. Specifically, the sound storage unit 1020 stores sound around the imaging device 100. Note that the recording time may be the time when voice recording starts or the time when recording ends.

  The feature sound extraction unit 1094 extracts a predetermined type of sound from the sound stored in the sound storage unit 1020. Specifically, the voice acquisition unit 1096 acquires the same type of voice as the voice extracted by the feature sound extraction unit 1094 from a voice database that stores a plurality of types of voice. For example, the feature sound extraction unit 1094 extracts music from the sound stored in the sound storage unit 1020. Then, the sound acquisition unit 1096 acquires the same type of music as the music extracted by the feature sound extraction unit 1094 from the music database 172 storing a plurality of music. In addition, the sound acquisition unit 1096 acquires the same type of environmental sound as the environmental sound extracted by the feature sound extraction unit 1094 from the environmental sound database 174. Note that the specific operation in which the sound acquisition unit 1096 acquires sound such as music or environmental sound is the same as the operation of the sound acquisition unit 696 described with reference to FIG.

  The output request acquisition unit 1048 acquires an output request for an image stored in the image storage unit 1010. The permissible time setting unit 1043 is a permissible time during which sound is extracted when the difference between the time when the output request acquisition unit 1048 acquires the output request and the image capturing time stored in the image storage unit 1010 is larger. Set a longer time. Then, the feature sound extraction unit 1094 extracts music from the sound recorded within the preset allowable time from the time when the image was captured. The output unit 1024 outputs the audio acquired by the audio acquisition unit 1096 and the image stored in the image storage unit 1010 in synchronization. Specifically, the output unit 1024 outputs the music or environmental sound acquired by the audio acquisition unit 1096 and the image stored in the image storage unit 1010 in synchronization.

  According to the output device 1040 of this embodiment, the captured image can be provided to the user 180 together with music that was played when the image was captured, for example, music that was popular at the time of imaging.

  FIG. 11 shows an example of the allowable time set by the allowable time setting unit 1043. For example, when an instruction to output an image captured at time t12 is received from the user 180 at time t13, the allowable time setting unit 1043 includes the time when the output is instructed and the time when the output image is captured. On the basis of the difference (t13−t12), an allowable range Δt52 for causing the feature sound extraction unit 1094 to output a sound for extracting music is determined. Then, the feature sound extraction unit 1094 among the voices stored in the voice storage unit 1020, from the voices recorded in the time range before or after Δt52 from time t12 (time t12−Δt52 to time t12 + Δt52). Extract music.

  Note that the feature sound extraction unit 1094 may extract the sound from the sound recorded between the time t12 before the time t12 and the time t12 before the time t12, or after the time t12 the time t12 after the time t12. Voices may be extracted from voices recorded up to the time.

  Further, the allowable time setting unit 1043 sets the allowable time to be larger as the difference between the time when the captured image stored in the image storage unit 1010 is captured and the time when the instruction to output is received is larger. In the example of FIG. 11, the allowable time setting unit 1043 is longer in time than the allowable range Δt52 when instructed at time t13 to output an image captured at time t11 prior to time t12. An allowable range Δt51 is set. Then, the feature sound extraction unit 1094 extracts music from the sound recorded within the time range from the time (t11−Δt51) to the time (t11 + Δt51).

  The allowable time setting unit 1043 may set a period obtained by dividing the time between the imaged time and the time when the output is instructed by a predetermined number as the allowable time. In this case, the feature sound extraction unit 1094 extracts music from the sound recorded during one day before and after the imaging time, for example, when outputting an image captured ten days ago. Further, when outputting an image captured ten years ago, the feature sound extraction unit 1094 extracts music from voices recorded for one year before and after the imaging time.

  According to the output device 1040 described above, for example, FM broadcast music heard in a car moving in a sightseeing spot, cable broadcasting music flowing in a shop at a sightseeing spot, etc. It is possible to appropriately determine the music that has been downloaded, download the music, and play it with the image. Further, the output device 1040 can download the environmental sound of the same type as the environmental sound of the place visited by the user 180 and reproduce it together with the image. Further, when outputting an image captured in the past, the output device 1040 selects a sound including music and environmental sound from sounds recorded in a wider time range including the imaged time. 180 can enjoy the image happily while remembering the most popular music at the time of imaging.

  FIG. 12 illustrates an example of a hardware configuration of a computer 1500 related to the output device 140, the imaging device 600, and the output device 1040. The computer 1500 includes a CPU peripheral unit having a CPU 1505, a RAM 1520, a graphic controller 1575, and a display device 1580 connected to each other by a host controller 1582, and a communication interface 1530 connected to the host controller 1582 by an input / output controller 1584. An input / output unit having a hard disk drive 1540 and a CD-ROM drive 1560, and a legacy input / output unit having a ROM 1510, a flexible disk drive 1550, and an input / output chip 1570 connected to the input / output controller 1584.

  The host controller 1582 connects the RAM 1520, the CPU 1505 that accesses the RAM 1520 at a high transfer rate, and the graphic controller 1575. The CPU 1505 operates based on programs stored in the ROM 1510 and the RAM 1520 and controls each unit. The graphic controller 1575 acquires image data generated by the CPU 1505 and the like on a frame buffer provided in the RAM 1520 and displays the image data on the display device 1580. Alternatively, the graphic controller 1575 may include a frame buffer that stores image data generated by the CPU 1505 or the like.

  The input / output controller 1584 connects the host controller 1582 to the hard disk drive 1540, the communication interface 1530, and the CD-ROM drive 1560, which are relatively high-speed input / output devices. The hard disk drive 1540 stores programs and data used by the CPU 1505 in the computer 1500. The communication interface 1530 communicates with the output device 140, the imaging device 600, or the output device 1040 via a network, and provides a program and data to the output device 140, the imaging device 600, or the output device 1040. The CD-ROM drive 1560 reads a program or data from the CD-ROM 1595 and provides it to the hard disk drive 1540 and the communication interface 1530 via the RAM 1520.

  The input / output controller 1584 is connected to the ROM 1510, the flexible disk drive 1550, and the relatively low-speed input / output device of the input / output chip 1570. The ROM 1510 stores a boot program executed when the computer 1500 is started up, a program depending on the hardware of the computer 1500, and the like. The flexible disk drive 1550 reads a program or data from the flexible disk 1590 and provides it to the hard disk drive 1540 and the communication interface 1530 via the RAM 1520. The input / output chip 1570 connects various input / output devices via a flexible disk drive 1550 and, for example, a parallel port, a serial port, a keyboard port, a mouse port, and the like.

  A program provided to the communication interface 1530 via the RAM 1520 is stored in a recording medium such as the flexible disk 1590, the CD-ROM 1595, or an IC card and provided by the user. The program is read from the recording medium, provided to the communication interface 1530 via the RAM 1520, and transmitted to the output device 140, the imaging device 600, or the output device 1040 via the network. The program transmitted to the output device 140, the imaging device 600, or the output device 1040 is installed and executed in the output device 140, the imaging device 600, or the output device 1040.

  The program installed and executed in the output device 140 causes the output device 140 to function as the output device 140 described with reference to FIGS. Further, the program installed and executed in the imaging apparatus 600 causes the imaging apparatus 600 to function as the imaging apparatus 600 described with reference to FIGS. Further, the program installed and executed in the output device 1040 causes the output device 1040 to function as the output device 1040 described with reference to FIGS. 10 and 11.

  The program shown above may be stored in an external storage medium. As the storage medium, in addition to the flexible disk 1590 and the CD-ROM 1595, an optical recording medium such as a DVD or PD, a magneto-optical recording medium such as an MD, a tape medium, a semiconductor memory such as an IC card, or the like can be used. Further, a storage device such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet may be used as a recording medium, and the program may be provided to the computer 1500 via the network.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. Various modifications or improvements can be added to the above embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

It is a figure which shows an example of an audio | voice provision system. 3 is a diagram illustrating an example of a block configuration of an output device 140. FIG. It is a figure which shows an example of the data which the description sound database 170 stores in a table format. It is a figure which shows an example of distribution of the position where the image was imaged. It is a figure which shows an example of distribution of the time when the image was imaged. FIG. 25 is a diagram illustrating an example of a block configuration of the imaging apparatus 600. It is a figure which shows an example of the data which the music database 172 stores in a table format. It is a figure which shows an example of the audio | voice and time range which the imaging device 600 records. It is a figure which shows an example of the procedure which acquires music. 5 is a diagram illustrating an example of a block configuration of an output device 1040. FIG. It is a figure which shows an example of allowable time. 2 is a diagram illustrating an example of a hardware configuration of a computer 1500. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Imaging device 140 Output device 150 Communication line 170 Explanation sound database 172 Music database 174 Environmental sound database 180 User 210 Image storage part 224 Output part 262 Sound acquisition part 278 Image selection part 282 Imaging area judgment part 284 Imaging period judgment part 286 Imaging position Distribution calculation unit 288 Number of imaging calculation unit 600 Imaging device 602 Imaging unit 650 Recording unit 660 Condition storage unit 670 Display unit 692 Mode setting unit 694 Feature sound extraction unit 696 Audio acquisition unit 698 Data storage unit 1010 Image storage unit 1020 Audio storage unit 1024 Output unit 1040 Output device 1043 Allowable time setting unit 1048 Output request acquisition unit 1060 Condition storage unit 1094 Feature sound extraction unit 1096 Audio acquisition unit 1044 Output time detection unit

Claims (15)

An imaging unit;
A recording unit for recording sound around the imaging unit;
A feature sound extraction unit that extracts a predetermined type of sound from the sound recorded by the recording unit;
A voice acquisition unit that acquires a voice of the same type as the voice extracted by the feature sound extraction unit from a voice database that stores a plurality of types of voice;
An imaging apparatus comprising: a data storage unit that stores the audio acquired by the audio acquisition unit and the image captured by the imaging unit in association with each other so as to be output in synchronization. The imaging according to claim 1, wherein the characteristic sound extraction unit extracts a predetermined type of sound from the sound recorded by the recording unit within a preset time from the time when the image capturing unit captures an image. apparatus. A display unit for displaying an image of light received by a light receiving element included in the imaging unit;
A mode setting unit that sets the imaging apparatus to an imaging mode that is an operation mode in which the display unit is displaying an image, or a non-imaging mode that is an operation mode in which the display unit is not displaying an image; Further comprising
The recording unit records sound around the imaging unit both when the mode setting unit is set to the imaging mode and when the mode setting unit is set to the non-imaging mode. The imaging device according to claim 2. The characteristic sound extraction unit includes a time set by the mode setting unit in the imaging mode, and is longer than a time set by the mode setting unit in the imaging mode within a preset time. The imaging apparatus according to claim 3, wherein a predetermined type of sound is extracted from the sound recorded by the recording unit. The voice database stores a plurality of music,
The feature sound extraction unit extracts music from the voice recorded by the recording unit,
The imaging apparatus according to claim 1, wherein the voice acquisition unit acquires the same music as the music extracted by the feature sound extraction unit from the voice database. The voice database stores a plurality of music by era,
The feature sound extraction unit extracts music from the voice recorded by the recording unit,
The imaging apparatus according to claim 1, wherein the voice acquisition unit acquires music of the same era as the music extracted by the feature sound extraction unit from the voice database. The voice database stores a plurality of music by genre,
The feature sound extraction unit extracts music from the voice recorded by the recording unit,
The imaging apparatus according to claim 1, wherein the voice acquisition unit acquires music of the same genre as the music extracted by the feature sound extraction unit from the voice database. A condition storage unit that preliminarily stores conditions for specifying each type of environmental sound extracted by the feature sound extraction unit;
The voice database stores a plurality of environmental sounds for each type of environmental sound,
The characteristic sound extraction unit extracts an environmental sound that matches the condition stored in the condition storage unit from the sound recorded by the recording unit,
The voice acquisition unit acquires the same kind of environmental sound as the environmental sound extracted by the feature sound extraction unit from the voice database,
The imaging apparatus according to claim 1, wherein the data storage unit stores the environmental sound acquired by the audio acquisition unit and the image captured by the imaging unit in association with each other so as to be output in synchronization. An imaging stage in which an image is captured by an imaging unit;
A recording stage for recording sound around the imaging unit;
A feature sound extraction step of extracting a predetermined type of sound from the sound recorded in the recording step;
A voice acquisition stage for acquiring a voice of the same type as the voice extracted in the feature sound extraction stage from a voice database storing a plurality of types of voice;
An imaging method comprising: a data storage step of storing the audio acquired in the audio acquisition step and the image captured by the imaging unit in association with each other so as to be output in synchronization. A program for an imaging apparatus that captures an image, wherein the imaging apparatus is
An imaging unit for capturing an image;
A recording unit for recording sound around the imaging unit;
A feature sound extraction unit for extracting a predetermined type of sound from the sound recorded by the recording unit;
A voice acquisition unit that acquires a voice of the same type as the voice extracted by the feature sound extraction unit from a voice database that stores a plurality of types of voice;
A program that functions as a data storage unit that stores the audio acquired by the audio acquisition unit and the image captured by the imaging unit in association with each other so as to be output in synchronization. An image storage unit for storing an image captured by the imaging device;
An audio storage unit for storing audio recorded by the imaging device;
A feature sound extraction unit that extracts a predetermined type of sound from the sound stored in the sound storage unit;
A voice acquisition unit that acquires a voice of the same type as the voice extracted by the feature sound extraction unit from a voice database that stores a plurality of types of voice;
An output device comprising: an output unit that outputs the audio acquired by the audio acquisition unit and the image stored in the image storage unit in synchronization. The image storage unit stores the imaging time of the image in association with the image,
The voice storage unit stores the recording time of the voice in association with the voice,
The output device according to claim 11, wherein the characteristic sound extraction unit extracts a predetermined type of sound from sound recorded within a preset allowable time from the time when the image was captured. An output request acquisition unit for acquiring an output request for an image stored in the image storage unit;
And a permissible time setting unit that sets the permissible time longer when the difference between the time when the output request acquisition unit acquires the output request and the image capture time of the image stored in the image storage unit is larger. The output device according to claim 12. An image storage stage for storing an image captured by the imaging device;
A voice storage step for storing voice recorded by the imaging device;
A feature sound extraction step of extracting a predetermined type of sound from the sound stored in the sound storage step;
A voice acquisition stage for acquiring a voice of the same type as the voice extracted in the feature sound extraction stage from a voice database storing a plurality of types of voice;
An output method comprising: an output step of outputting the sound acquired in the sound acquisition step and the image stored in the image storage step in synchronization. A program for an output device for outputting an image, wherein the output device is
An imaging unit for capturing an image;
A recording unit for recording sound around the imaging unit;
A feature sound extraction unit for extracting a predetermined type of sound from the sound recorded by the recording unit;
A voice acquisition unit that acquires a voice of the same type as the voice extracted by the feature sound extraction unit from a voice database that stores a plurality of types of voice;
A program that functions as a data storage unit that stores the audio acquired by the audio acquisition unit and the image captured by the imaging unit in association with each other so as to be output in synchronization.

Images