JP2011205296A - Apparatus and program for image decoration - Google Patents

Abstract

PROBLEM TO BE SOLVED: To more effectively decorate images than documents consisting of only character information.SOLUTION: When an image decoration mode is already configured, if image data is generated by capturing an image, object recognition is performed through image analysis. When an object is specified, decoration content is extracted from a decoration database based on it, and then the image data is decorated with the extracted decoration. Based on the decorated image data, an image file is structured and recorded into a recording medium.

Description

  The present invention relates to an image decoration device and an image decoration program capable of decorating an electronic image.

  Read information (eg, user registration memo, storage folder name, information at the time of recording, playback frequency, preference information, etc.) associated with the image or sound data, and generate a sentence based on the read information There is known a speech synthesizer that synthesizes and outputs a synthesized sentence (for example, see Patent Document 1).

JP 2006-23592 A

  In the apparatus of Patent Document 1, since text is simply extracted by extracting character information attached to image data, the created text lacks interest.

  The image decoration device according to the present invention is based on image data or information associated with the image data, estimating means for estimating the subject or shooting situation of the image data, and the subject or shooting situation estimated by the estimating means. The image processing apparatus includes: a determination unit that determines suitable image decoration content; and a decoration unit that applies the decoration of the content determined by the determination unit to the image data and outputs the image data.

  According to the present invention, it is possible to add more interesting decoration to an image compared to a case where a sentence is simply created from character information, which can give the image viewer surprise and excitement. You can shoot while enjoying more than ever.

FIG. 3 is a control block diagram of the digital camera according to the first embodiment. The figure which shows the example of a display of the image which gave the image decoration. The figure which shows the other example of a display of the image which gave the image decoration. The figure which shows the other example of a display of the image which gave the image decoration. The figure which shows the other example of a display of the image which gave the image decoration. 6 is a flowchart illustrating a processing procedure of the camera according to the first embodiment. The figure which shows the other example of a display of the image which gave the image decoration. The system block diagram in 2nd Embodiment. The figure which shows the other example of a display of the image which gave the image decoration. The flowchart which shows the process sequence of PC in 2nd Embodiment.

-First embodiment-
An embodiment when the present invention is applied to a digital camera will be described with reference to FIGS.
FIG. 1 is a block diagram of a digital camera according to this embodiment. The subject luminous flux that has passed through the photographic lens 1 is imaged by the imaging device 2, and the imaging signal is input to the image processing unit 3. The image processing circuit 3a constituting the image processing unit 3 performs various processes on the input imaging signal to generate image data. The image data is displayed on a liquid crystal monitor 4 provided on the back of the camera or the like through processing by the display circuit 3b. When the shooting mode is set, the above imaging and image display are repeated, and so-called live view display (through image display) is performed. When a shooting instruction is given, the image is taken again, and the image data generated thereby is recorded on the recording medium 5 such as a memory card by the recording / reproducing circuit 3c as an image file that can be handled by a computer.

  The image file to be recorded is configured by combining the above-described image data (compressed as necessary) constituting the image main body and metadata such as shooting date / time, shooting location, and shooting conditions. The shooting conditions include information such as a used scene mode in addition to the aperture value, shutter speed, and ISO sensitivity.

  In the reproduction mode, the image data recorded on the recording medium 5 can be read out by the recording / reproduction circuit 3c and displayed on the liquid crystal monitor 4 through the processing by the image processing circuit 3a and the display circuit 3b.

  The CPU 6 controls the image processing unit 3 and a circuit (not shown) in response to an input from the operation unit 7. The operation unit 7 includes a power button, a release button, various operation members used for reproduction operation, information input, and the like. The GPS receiver 8 detects camera position information (latitude and longitude information), and the CPU 6 can embed position information (shooting position information) detected at the time of shooting in the image file as the metadata. Reference numeral 9 denotes a memory including a ROM and a RAM.

Next, image decoration in the present embodiment will be described.
Image decoration here refers to inserting onomatopoeia, onomatopoeia, mimetic words, and the like into the image data, and inserting non-characters such as marks and lines suitable for the image. The decoration content is automatically determined by the CPU 6 based on image data or metadata.

  In order to decorate an image based on image data, the memory 9 needs to have a function for recognizing (estimating) a subject in advance. The subject recognition function is a function of analyzing image data and estimating a subject from the shape, color, and the like, and for example, a technique such as template matching can be used. When using template matching, a database storing a plurality of types of subject templates is stored in the memory 9 in advance.

  The memory 9 also stores a decoration database for extracting decoration contents suitable for the image. This database associates various subjects with character strings representing the sounds, calls, etc. emitted by the subjects as decoration contents. For example, “Bazai” or “Automobile” for “Sea” On the other hand, “Buroro”, “Thunder” “Pika”, “Fireworks” “Dokan”, “Cat” “Nyan”, “Dog” “Wan”, For example, “Hello!” For “soccer game”.

  The CPU 6 searches the decoration database based on the result of the subject recognition, extracts the decoration contents, and applies the decoration to the image data based on it. An image based on the decorated image data can be displayed on the liquid crystal monitor 4.

  FIG. 2 is a display example of image data obtained by photographing “the sea”. In this case, the CPU 6 estimates that the subject is “the sea” due to the large proportion of the blue system in the lower part of the image, for example, and searches the decoration database using “the sea” as a keyword. As a result of the search, a character string “Zaza” is extracted, and this character string is inserted into the original image data. When it is determined that the subject is “mountain”, a character string such as “Yaho” is inserted as shown in FIG.

  FIG. 4 is a display example of image data obtained by photographing “cat”. In this case, the CPU 6 estimates that the subject is “cat” based on the shape and color of the subject, and searches the decoration database using “cat” as a keyword. As a result of the search, a character string “Nyan” is extracted, and this character string is inserted into the original image data.

  Similarly, FIG. 5 is a display example of image data obtained by photographing a “race car”. In this case, the CPU 6 estimates that the subject is a “race car” by template matching or the like, and searches the decoration database using “race car” as a keyword. As a result of the search, a character string “VOOOON” is extracted, and this character string is inserted into the original image data.

In the example of FIG. 5, not only a character string but also a plurality of radial straight lines are inserted around a “race car” recognized as a main subject. Such visual effects are also extracted from the decoration database simultaneously with the character string. By applying visual effects other than character strings, it is possible to highlight the main subject and emphasize the movement of the subject.
The recognition of the main subject will be described later.

  Here, in a camera having a known moving object prediction function, image decoration (visual effect) can be applied based on the prediction result. Although the moving object prediction is originally used for focus adjustment, since the CPU 6 can recognize the moving direction of the subject by the moving object prediction, a streamline (for example, a direction opposite to the moving direction from the main subject) can be known. It is conceivable to insert a plurality of lines extending into the image. In this case, the decoration can be applied without performing the database search as described above.

  A known face recognition technique or facial expression recognition technique may be used as a subject recognition technique. First, face recognition is performed on image data. If a face is recognized, facial expression recognition is performed. For example, if it is recognized that it is a “smile,” a character string such as “smile” or “niyaya”, or if it is recognized as a “crying face” is extracted from the decoration database, and the image data is Inserted. For facial expression recognition, for example, a technique described in Japanese Patent Application Laid-Open No. 2001-51338 can be used.

  Here, when the decoration is applied, it is necessary to prevent the main subject from being covered with the decoration (character string or the like). Therefore, the CPU 6 recognizes the main subject intended by the photographer based on the in-focus position (focus area), the luminance distribution and color information of the image data, or the above-described subject recognition result, and is out of the main subject. Place a string at a position.

In order to make the decoration stand out, the color of the character string is determined based on the color of the part to be decorated, the font and character size, hiragana, katakana or romaji are determined according to the contents of the character string, etc. It is desirable to determine the character size, horizontal writing or vertical writing according to the size and shape of the portion to be applied.
Furthermore, in a camera equipped with the GPS receiver 8, a character string as a decoration may be configured in the language or dialect of the country or region where the image was taken based on the shooting position information. For example, for image data shot in an English-speaking region, it is conceivable to insert a character string in English or insert an English expression as a Japanese character string.

FIG. 6 is a flowchart showing an example of a processing procedure at the time of shooting including image decoration. This is an example of applying image decoration immediately after shooting.
In response to the operation of the release button, the CPU 6 activates this program and performs image capturing in step S1 to obtain image data. In step S2, it is determined whether or not the image decoration mode is set. The setting / cancellation of the image decoration mode can be switched on, for example, a menu screen.

  If the image decoration mode is not set, the process proceeds to step S7, and an image based on the image data is displayed on the liquid crystal monitor 4. In step S8, the image data is compressed and metadata is added to form an image file. In step S9, the image file is recorded on the recording medium 5 and the process ends. The image displayed in step S7 is turned off after a predetermined time (several seconds).

  On the other hand, if it is determined in step S2 that the image decoration mode is set, the process proceeds to step S3 to perform the above-described subject recognition. In step S4, it is determined whether or not the subject has been identified in step S3. For example, a subject that is not registered in the database for subject recognition cannot be specified, and even if it is registered, the subject may not be specified for some reason. If the subject cannot be specified, the process proceeds to step S10, a display indicating that the image decoration cannot be performed is displayed on the liquid crystal monitor 4, and the process proceeds to step S7.

  If the subject can be identified, the process proceeds to step S5, and the above-described decoration database search is performed to extract the contents (character string) to be used as decoration. At this time, the color and font of the character string are also determined. In step S6, the extracted decoration content is applied to the image data, and the process proceeds to step S7. In this case, an image based on the decorated image data is displayed on the liquid crystal monitor 4 (step S7), and an image file is constructed and recorded using the decorated image data (steps S8 and S9). Note that both a decorated image file and an undecorated image file may be recorded.

  Although the character string is inserted into the image as a decoration as described above, a mark or a picture may be inserted instead of the character string. For example, based on the result of facial expression recognition described above, it may be possible to insert a mark or a pattern suitable for the recognized facial expression into an image. FIG. 7 shows an example of this. Since the “angry” facial expression of the subject person is detected, the CPU 6 searches the database using “angry” as a keyword, extracts the angry mark MK1, and inserts it into the image. When “smile” is detected, for example, a pattern of “flower” is inserted.

Further, in the above, image decoration is performed at the time of shooting and the decorated image data is recorded, but decoration is not performed at the time of shooting, and the image file already recorded in the recording medium 5 is read out, and the read image file The image data may be decorated and recorded again after the decoration.
In addition, although it demonstrated with the digital camera, this invention is applicable also to the mobile telephone with a camera function etc., for example.

-Second Embodiment-
A second embodiment of the present invention will be described with reference to FIGS.
This embodiment is different from the previous embodiment in that image decoration is performed by a personal computer (hereinafter referred to as a PC) and that image decoration is performed based on metadata and audio data of an image file. Note that the functions of the present embodiment may be provided in the digital camera as described above, or conversely, the functions of the previous embodiment may be provided in the PC.

  FIG. 8 is a system configuration diagram in the present embodiment. The PC 20 includes a CPU 21, a storage unit 22 including a RAM, a hard disk, and the like, a display unit 23 such as a liquid crystal display, an operation unit 24 such as a mouse and a keyboard, and a communication unit 25 for performing network communication. Yes. In the hard disk, transfer software for capturing an image file from a digital camera, image processing software for processing or reproducing the captured image file, and an external server (40, 50, etc.) via the communication network 30 Communication software for access is stored in advance. The server 40 provides a service that provides weather information of each location, and the server 50 provides a service that provides information related to events (such as sporting events and fireworks events) that are performed in each location.

  Here, some digital cameras can record ambient sound during recording to create sound data and record it in association with an image file. The CPU 21 captures an image file from the digital camera using the transfer software. If there is audio data associated with the image file, the CPU 21 also captures the audio data.

The image decoration in this embodiment will be described.
When the target image file has shooting position information (latitude and longitude information) as metadata, the CPU 21 estimates the weather at the time of shooting based on the shooting position information and shooting date / time information. This is done by sending the shooting position information and shooting date / time information to the server 40 via the communication network 30 and acquiring weather information at the shooting date / time of the area including the shooting position information. Then, the decoration content is extracted from the decoration database based on the acquired weather information and applied to the image data in the image file.

  FIG. 9 shows a display example of an image decorated based on weather information. In this image, a character string “Jimejime” representing an uncomfortable state when temperature and humidity are high is decorated as a decoration. . In this example, a font that further increases discomfort is used. When the temperature is higher, a character string such as “hot” is inserted.

  Similarly, the CPU 21 estimates the captured image by sending the shooting position information and shooting date / time information to the server 50 and acquiring event information performed at the shooting location on the shooting date / time. be able to. Then, based on the estimation result, the decoration content is extracted from the decoration database and applied to the image data in the image file. For example, if it is determined that a baseball game has been held, a character string such as “Kakein” is inserted into the image data, and if it is determined that a fireworks display has been held, “ A character string such as “don” or “tamaya-kaguya-” is inserted.

  Further, image decoration can be applied based on scene mode information as metadata. Many digital cameras have multiple scene modes, and when the photographer selects one of the scene modes according to the scene they want to shoot, the camera automatically sets the shooting conditions such as the exposure value suitable for that scene. Set and take a picture. Scene modes include “Landscape mode”, “Portrait mode”, “Sport mode”, “Night scene mode”, “Sunset mode”, etc. Use mode information (scene mode information) is embedded in the image file as the metadata.

  The CPU 21 reads the scene mode information from the image file, so that the subject and the shooting situation can be estimated, and based on this, the decoration content is extracted from the decoration database and applied to the image data. For example, in the case of an image taken in the “sport mode”, a radial straight line as described in FIG. 4 is inserted, or in the case of an image taken in the “sunset mode”, the “crow” is a crow call. It is conceivable to insert a character string such as

  Furthermore, with respect to image data associated with audio data, the audio data can be analyzed and decoration can be applied based on the volume or the like. For example, if the sound is low, a character string such as “Scene” is inserted, and if the sound is high, a character string such as “Wai Wai” or “Gaya Gaya” is inserted.

  FIG. 10 is a flowchart illustrating an example of a processing procedure for image decoration by the CPU 21. This processing can be incorporated as one function of the image processing software, for example. When the user selects an image file to be decorated, the CPU 21 displays an image based on the image data of the image file on the display unit 23. Thereafter, when the user performs a predetermined operation, the CPU 21 activates the process of FIG. To do.

  The CPU 21 first reads metadata from the image file in step S21, and determines whether or not shooting position information is included in the metadata in step S22. If step S22 is negative, the process proceeds to step S28. If the determination is positive, in step S23, the shooting position information and the shooting date / time information, which are metadata, are sent to the server 40 via the communication network 30, and the shooting location at the shooting date / time is determined. Get weather information. In step S24, a decoration database search is performed based on the acquired weather information to acquire decoration contents.

  In step S24, shooting position information and shooting date / time information, which are metadata of the image file, are sent to the server 50 via the communication line network 30, and acquisition of event information performed at the shooting location at the shooting date / time is attempted. In step S26, it is determined whether or not event information has been acquired. If the determination is negative, the process proceeds to step S28. If the determination is affirmative, in step S27, a decoration database search is performed based on the acquired event information to acquire the decoration contents. .

  In step S28, it is determined whether or not scene mode information exists as meter data of the image file. If the determination is negative, the process proceeds to step S30. If step S28 is affirmed, the process proceeds to step S29, where a decoration database search is performed based on the scene mode information, and the decoration content is extracted.

  In step S30, it is determined whether or not the audio file is associated with the image file. If the result is negative, the process proceeds to step S32. If step S30 is affirmed, the process proceeds to step S31, where a decoration database search is performed based on the audio data to extract decoration contents.

  In step S32, it is determined whether or not at least one decoration content has been extracted. If the determination is negative, the process proceeds to step S38, a display indicating that image decoration cannot be performed is displayed on the display unit 23, and the process is terminated. For example, if there is no shooting position information and scene mode information in the metadata and no audio data is associated, step S32 is negative.

  When step S32 is affirmed, it progresses to step S33 and the image decoration content is determined. When a plurality of image decoration contents are extracted (when decoration contents are extracted in two or more of steps S24, S27, S29, and S31), the CPU 21 selects one of them by some algorithm. To do. All decoration contents may be displayed so that the user can select them. Or you may make it give all the decorations extracted.

  In step S34, the decoration content determined in step S33 is applied to the image data of the image file, and the image after decoration in step S35 is displayed on the display unit 23. In step S36, the user is asked whether or not to save the decorated image. When the “YES” operation is performed, the image file is stored, and when the “NO” operation is performed, the process ends without saving.

2 Image sensor 3 Image processor 6 CPU (Camera)
20 PC (personal computer)
21 CPU (PC)

Claims (8)

Based on image data or information associated with the image data, estimating means for estimating the subject of the image data or the shooting situation;
Determining means for determining image decoration contents suitable for the subject or shooting situation estimated by the estimating means;
An image decoration apparatus comprising: decoration means for applying the decoration of the content determined by the determination means to the image data and outputting the image data.   The image decoration device according to claim 1, wherein the estimation unit estimates the subject or the shooting state by performing an image analysis on the image data.   The image decoration device according to claim 1, wherein the estimation unit estimates the shooting situation based on shooting position information associated with the image data.   The image decoration device according to claim 1, wherein the estimation unit estimates the shooting state based on audio data at the time of shooting associated with the image data.   The image decoration device according to claim 1, wherein the determination unit determines the decoration content by performing a database search based on an estimation result of the estimation unit.   The decoration content is an onomatopoeia, an onomatopoeia, or an imitation word related to a subject, and the decoration means decorates the onomatopoeia, onomatopoeia, or an imitation word so that it is displayed as a character string in an image. The image decoration apparatus according to claim 1.   The image decoration apparatus according to claim 1, further comprising an imaging unit that performs imaging to generate the image data. On the computer,
Based on image data or information associated with the image data, a process for estimating the subject of the image data or the shooting situation;
A process of determining image decoration content suitable for the estimated subject or shooting situation;
An image decoration program for executing a process of applying the decoration of the determined content to the image data and outputting the image data.

Images