JP5320913B2 - Imaging apparatus and keyword creation program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically associate a proper keyword with an image file. <P>SOLUTION: An object is imaged to generate image data, and voice data at the imaging are acquired. The color distribution or luminance distribution of the image data is checked by using an image analyzing function, and scene candidates are extracted from a table in which the color distribution or luminance distribution is associated with the scene. On the other hand, frequency or waveform of the voice data are checked by using a voice analyzing function, and the scene candidates are extracted from the table in which the frequency and waveform are associated with the scene. The extracted scene candidates are compared with each other so that the common candidate can be used as a keyword. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to an imaging device and a keyword creation program that generate keywords for image data using sound data.

There is an apparatus that can acquire image data by imaging, acquire surrounding sound data at the time of imaging, determine the sound content from the sound data, and assign the sound content as a keyword to the image data. For example, when a wave sound is acquired at the time of imaging, “wave sound” is assigned to the metadata area of the image file as a keyword (see, for example, Patent Document 1).
The assigned keyword is used, for example, as a tag in image search and also for classification of image files.

JP 2005-346440 A (paragraph 0053, etc.)

  However, it is difficult to accurately specify the content of the sound only with the sound data, and there is a possibility that erroneous information may be acquired as a keyword in the method of Patent Document 1. For example, the sound of a wave is very similar to the sound of a hustle and a “wave sound” may be added to image data taken in a city far from the sea.

An imaging apparatus according to the present invention associates an imaging unit that images a subject to generate image data, a sound acquisition unit that acquires sound data at the time of imaging, a result of analyzing a color distribution of the image data, and a keyword Storage means for storing the first table, a second table in which the result of analyzing the frequency of the sound data and the keywords are associated, and the color distribution of the generated image data is analyzed to correspond to the first table. A keyword is extracted, a frequency corresponding to the acquired sound data is analyzed, a corresponding keyword is extracted from the second table, a keyword common to each extracted keyword is determined, and the determined keyword is used as image data. Recording means for recording in association with each other.
The keyword creating program according to the present invention associates a computer with a means for capturing an image of a subject to generate image data, a means for acquiring sound data at the time of image capturing, a result of analyzing a color distribution of image data, and a keyword. Means for storing a first table and a second table in which the result of analyzing the frequency of sound data and a keyword are associated; and analyzing the color distribution of the generated image data and extracting the corresponding keyword from the first table And analyzing the frequency of the acquired sound data to extract a corresponding keyword from the second table, determining a keyword common to each extracted keyword, and recording the determined keyword in association with the image data. It functions as a means.

  According to the present invention, it is possible to automatically associate an accurate keyword with an image file.

An embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a control block diagram of the digital camera in the present embodiment. The subject luminous flux that has passed through the photographic lens 1 is imaged by the imaging device 2, and the imaging signal is input to the image processing unit 3. The image processing circuit 3a constituting the image processing unit 3 performs various processes on the input imaging signal to generate image data. The image data is displayed on a liquid crystal monitor 4 provided on the back of the camera or the like through processing by the display circuit 3b. When the shooting mode is set, the above imaging and image display are repeated, and so-called live view display (through image display) is performed.

  When an imaging instruction is given, imaging is performed again, and the generated image data is recorded as an image file on the recording medium 5 such as a memory card via the recording / reproducing circuit 3c. In general, an image file is created by combining data (so-called image data) constituting an image main body and metadata as additional information.

  Further, the camera is provided with a microphone 7 and a speaker 8, the surrounding sound input from the microphone 7 at the time of imaging is acquired as sound data, and a sound file created based on the sound data is recorded in association with the image file. can do. Hereinafter, an image file associated with a sound file is referred to as an “image file with sound”.

  When the camera is set to the playback mode, the image file recorded on the recording medium 5 can be read out by the recording / playback circuit 3c and displayed as an image on the liquid crystal monitor 4 through processing by the image processing circuit 3a and the display circuit 3b. . As for the image file with sound, the corresponding sound data can be reproduced from the speaker 8 in synchronization with the image reproduction.

  The CPU 6 controls an image processing unit and a circuit (not shown) in response to an input from the operation unit 9. The operation unit 9 includes a power button, a release button, various operation members used for reproduction operation, information input, and the like.

  Incidentally, in order to support search and classification of image files, a keyword (tag) is associated with each image file and stored. The association is made, for example, by describing a keyword as character information in the metadata area of each image file (generally, it is said that the keyword is embedded in the image file). The camera of the present embodiment can automatically generate a keyword by analyzing image data and sound data for an image file with sound, and embed the keyword in the image file.

  In order to realize this automatic keyword generation, the camera examines, for example, the color distribution and luminance distribution of image data using an image analysis function, and extracts scene candidates from a table in which these are associated with scenes. On the other hand, the sound analysis function is used to check the frequency, waveform, etc. of the sound data, and a scene candidate is extracted from a table in which these are associated with the scene. Each extracted scene candidate is compared and a common one is used as a keyword.

  As an example, it is assumed that there is an image file with sound taken in a sea bathing. In the image analysis, since the ratio of blue is large in the image, “blue sky”, “sea” and the like are extracted as scene candidates. On the other hand, according to sound analysis, “busy”, “sea” (wave sound), etc. are extracted as scene candidates from the frequency and waveform of sound. In this case, since “sea” is common to both extraction candidates, the keyword is finally “sea”.

  As another example, it is assumed that there is an image with sound obtained by shooting a fireworks display. According to the image analysis, since there are a plurality of bright points in the dark as a whole, “night view” and “fireworks” are extracted as scene candidates. According to sound analysis, “explosion”, “fireworks” and the like are extracted from the explosion sound. Therefore, the keyword is “fireworks”.

  As described above, since the keyword is created from both the image data and the sound data, an accurate keyword can be obtained as compared with the case where the keyword is created only from the sound data.

  In order to deal with many scenes, the information in the table needs to be enriched. For example, it is desirable that a camera maker publishes new information on a website or the like as needed, and allows the user to download it and add it to the table or change it.

  Generation of keywords and embedding in an image file may be performed automatically at the time of shooting, or may be performed on an image file with sound that has already been shot. In the former case, an “automatic keyword assignment mode” may be provided as the imaging mode.

  FIG. 2 shows an example of a processing procedure when the automatic keyword assignment mode is set. When the release button is fully pressed, the CPU 6 activates this program, captures and records in step S1, and acquires image data and sound data, respectively. Image analysis and sound analysis are performed in steps S2 and S3, respectively, and an audio file is generated from the sound data in step S4. In step S5, keywords are extracted as described above based on the analysis results in steps S2 and S3. In step S6, the extracted keyword is described in the metadata area, and the metadata and image data are combined to generate an image file. In step S7, the image file and the sound file are associated and recorded on the recording medium 5.

  Here, when the image file and the sound file are recorded, they may be automatically stored in a folder named after the keyword. For example, if the extracted keyword is “sea”, if there is no folder named “sea” in the storage medium 5, it is created and the file is stored in the “sea” folder. Thereafter, a file in which “sea” is extracted as a keyword is stored in the “sea” folder. As a result, file classification can be automated.

  In the automatic keyword assignment mode, keywords are generated from image data and sound data, but in the case of person shooting, it is possible to ask a person who is the subject to speak a word, extract the keyword from the word and attach it to the file. Conceivable. This gives an accurate keyword. Hereinafter, a mode that enables this is referred to as a “voice keyword assignment mode”.

  The camera has a voice recognition function, a speaker recognition function, and a face recognition function. The voice recognition function is a function that recognizes the utterance content by analyzing voice data uttered by a person. Speaker recognition is a function that extracts features of voice (voice print, etc.) from voice data and searches a database for a person who utters the voice. This type of database is provided in the camera so that personal information and voice characteristics for a plurality of people can be registered in association with each other. The face recognition function detects the position and size of a person's face in image data, and is generally used to focus and expose the face portion.

  FIG. 3 shows a processing procedure when the voice keyword assignment mode is set. When the release button is fully pressed, the CPU 6 activates this program and performs image capturing and recording in step S11 to acquire image data and audio data, respectively. Note that the photographer requests the person who is the subject to speak the keyword prior to imaging. The camera may request the utterance of the keyword by voice. The subject person may say the date, shooting location, what he / she is doing now, and so on.

  In step S12, face recognition is performed, and it is determined whether or not a face is recognized in step S13. If the determination is affirmative, the process proceeds to step S14, where “portrait” is set as one of the keywords. In this mode, a normal face should be recognized, but if not recognized, the process proceeds to step S21.

  In step S15, speech recognition is performed. If it is determined in step S16 that the recognition has been successful, keywords are extracted based on the result of speech recognition in step S17. In step S18, speaker recognition is performed to identify the person who uttered the voice. In other words, it identifies who the subject person is. If it is determined in step S19 that the identification is successful, person information is used as a keyword in step S20. The person information is registered in the database and is, for example, the name of the person.

  In step S21, an audio file is generated based on the audio data. In step S22, the keywords acquired in steps S14, S17, S20, etc. are described in the metadata area, and the image file is combined with the metadata and the image data. Generate. Note that if a face is not recognized in the image data, the keyword cannot be acquired. In this case, the keyword may be acquired by the method shown in FIG.

  As described above, the keyword embedded in the image file can be used as a search tag, used for classification of the image file, and can be displayed as a character or a comment at the time of image reproduction. FIG. 4 shows an example of the display, and FIG. 4A shows an example in which characters are displayed in a so-called balloon. The keywords “June 10” and “harvest harvest” are both words uttered by a person in the image at the time of imaging. FIG. 4B shows an example in which characters are displayed as a title. Note that it is desirable that the number of characters, font, character color, and background color in the character display can be selected by the user according to preference.

  5 and 6 show examples of image display when a plurality of persons in conversation are captured and recorded. In such a case, by recognizing the correspondence between the speaker and the content of the utterance, that is, which person uttered which word by the above speech recognition and speaker recognition, and obtaining the time from the camera built-in clock, The camera can automatically grasp who said what and when. The grasped information is recorded in association with the image file, and when the image file is reproduced, the content of the conversation can be displayed like a minutes. FIG. 5 shows an example in which conversations are arranged in a time series, and FIG. 6 shows an example in which each person is grouped. It is desirable that both display modes can be switched by a simple operation.

  Further, in addition to associating the person with the conversation, if it is known who each person in the image is, it is possible to specify which person in the image has spoken which word. Identification of a person in an image can be realized by extracting feature information of each person's face by face recognition and performing a database search based on the extracted feature information. This kind of database may be provided in the camera so that personal information and facial feature information for a plurality of people can be registered in association with each other.

  Then, for example, the leftmost person in the image is found to be “A”. On the other hand, it is found from the above voice recognition and speaker recognition that “XXXXXX” is “A”. Then, it can be grasped that the person A located at the leftmost position in the image said “XXXXXXX”. If such correspondence is recorded in correspondence with the image file, for example, when displaying an image using a balloon as shown in FIG. 4A, the leftmost person says “XXXXXXX” The position and orientation of each balloon can be set so that the person can say “YYY”. For example, “A: XXXXXX” or “B: YYY” may be displayed in the balloon.

  4 to 6 may be realized by a program installed in advance when, for example, an image file is taken into an external device (for example, a personal computer). The keyword may be created by an external device. In this case, the program incorporated in the external device inputs an image file with sound or an image file with sound from, for example, a camera, and creates a keyword in the same manner as described above based on information read from the metadata section. That's fine.

  Although the above has been described with respect to a still image file, a moving image file may be used.

The control block diagram of the camera in one Embodiment of this invention. The flowchart which shows an example of the process sequence at the time of automatic keyword provision mode setting. The flowchart which shows an example of the process sequence at the time of voice keyword provision mode setting. The figure which shows the example of a display of the image imaged in the audio | voice keyword provision mode. The figure which shows the other example of a display of the image imaged in the audio | voice keyword provision mode. The figure which shows the other example of a display of the image imaged in the audio | voice keyword provision mode.

Explanation of symbols

2 Image sensor 3 Image processing unit 4 Liquid crystal monitor 6 CPU
7 Microphone 8 Speaker

Claims (5)

Imaging means for imaging a subject and generating image data;
Sound acquisition means for acquiring sound data at the time of imaging;
Storage means for storing a first table in which the result of analyzing the color distribution of the image data is associated with a keyword, and a second table in which the result of analyzing the frequency of the sound data is associated with a keyword;
Analyzing the color distribution of the generated image data to extract the corresponding keyword from the first table , analyzing the frequency of the acquired sound data to extract the corresponding keyword from the second table, A determination means for determining a keyword common to each extracted keyword ;
An image pickup apparatus comprising: a recording unit that records the determined keyword in association with the image data . The imaging device according to claim 1,
The first table stores a result of analyzing a luminance distribution of the image data and a keyword in association with each other. In the imaging device according to any one of claims 1 and 2,
The image recording apparatus, wherein the recording unit records the image data in a folder corresponding to the determined keyword. Computer
Means for imaging a subject and generating image data;
Means for acquiring sound data at the time of imaging;
Means for storing a first table associating a result of analyzing the color distribution of the image data with a keyword, and a second table associating a result of analyzing the frequency of the sound data with a keyword;
Analyzing the color distribution of the generated image data to extract the corresponding keyword from the first table, analyzing the frequency of the acquired sound data to extract the corresponding keyword from the second table, Means for determining keywords common to each extracted keyword,
A keyword creation program for causing the determined keyword to function as means for recording the keyword in association with the image data . In the keyword creation program according to claim 4,
The keyword creating program, wherein the recording means records the image data in a folder corresponding to the determined keyword.

Images