JP2013140559A - Image processing device, imaging device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of imparting more flexible character information to a captured image.SOLUTION: An image processing device 1 comprises: an image input section 10 that inputs a captured image; a storage section 90 that stores, as a text template in which a word is inserted into a predetermined blank column part to complete a text, a person image template to be used in preparation of a text for a person image in which a person is a photographic subject and a landscape image template to be used in preparation of a text for a landscape image in which a landscape is a photographic subject; a determination section 20 that determines whether or not the captured image is a person image or landscape image; and a text preparation section 30 that reads the text template of any of the person image template or landscape image template from the storage section 90 based on the determination result by the determination section 20 for the captured image and inserts words into the blank column part in accordance with an amount of features or image-capturing conditions of the captured image to prepare a text for the captured image.

Description

  The present invention relates to an image processing device, an imaging device, and a program.

  Conventionally, character information such as a birthday person name corresponding to an imaging date and an event name corresponding to an imaging date is given to a captured image by registering a birthday of the specific person, an event date and the like in advance. A technique is disclosed (for example, see Patent Document 1).

JP-A-2-303282

  However, the conventional technology has a problem that only character information registered in advance by the user can be added to the captured image. In view of the problems of the prior art, an object of the present invention is to provide a technique that can give more flexible character information to a captured image.

  In order to solve the above problems, an image processing apparatus according to an aspect of the present invention includes an image input unit that inputs a captured image, and a sentence template that completes a sentence by inserting a word into a predetermined blank part. A storage unit for storing a person image template used for creating a sentence for a person image that is a subject, a landscape image template used for creating a sentence for a landscape image whose landscape is a subject, and the captured image, A determination unit that determines whether the image is a person image or the landscape image, and the sentence template that is either the person image template or the landscape image template according to a determination result by the determination unit for the captured image Is read from the storage unit, and the blank portion of the read text template corresponds to the feature amount or the imaging condition of the captured image. It was inserted words, characterized in that it comprises a sentence creation unit for creating a sentence with respect to the captured image.

  In order to solve the above problems, an image processing apparatus according to another aspect of the present invention includes an image input unit to which a captured image is input, a feature amount of the captured image, and an imaging condition of the captured image. A determination unit that determines a text corresponding to the first type, a determination unit that determines whether the captured image is a first type image or a second type image different from the first type, and the first type A storage unit that stores a first syntax that is a syntax of a sentence used for the second type and a second syntax that is a syntax of a sentence that is used for the second type, and the captured image is the image of the first type When the determination unit determines that the sentence of the first syntax is created using the text determined by the determination unit, and the determination unit determines that the captured image is the second type image The text determined by the determination unit Characterized in that it comprises a sentence creation unit for creating a sentence of the second syntax using preparative.

  In addition, an imaging apparatus according to another aspect of the present invention includes an imaging unit that captures a subject and generates a captured image, and a sentence template that completes a sentence by inserting a word into a predetermined blank part and a person is the subject. A storage unit that stores a person image template used for creating a sentence for a certain person image and a landscape image template used for creating a sentence for a landscape image whose scenery is a subject, and the captured image includes the person image. The sentence template, which is either the person image template or the landscape image template, is determined according to a determination result by the determination unit that determines whether the image is a landscape image or the landscape image. A word corresponding to the feature amount or the imaging condition of the captured image is read from the storage unit and the blank part of the sentence template that has been read out. Input to characterized in that it comprises a sentence creation unit for creating a sentence with respect to the captured image.

  A program according to another aspect of the present invention includes a person image template used for creating a sentence for a person image in which a person is a subject as a sentence template for completing a sentence by inserting a word into a predetermined blank space. An image input step of inputting a captured image to a computer of an image processing apparatus including a storage unit for storing a landscape image template used for creating a sentence for a landscape image in which the landscape is a subject; and The sentence template of either the person image template or the landscape image template according to a determination step for determining whether the image is a person image or the landscape image, and a determination result of the determination step for the captured image Is read from the storage unit, and the blank part of the read sentence template Characterized in that to execute the sentence generating step of generating a sentence by inserting a word corresponding to the feature amount or the imaging condition of the image the image with respect to the captured image.

  An image processing apparatus according to one aspect of the present invention includes a determination unit that determines a character having a predetermined meaning from a captured image, and whether the captured image is a person image or an image different from the person image. A storage unit that stores a determination unit that determines a first syntax that is a sentence syntax used for the person image, and a second syntax that is a sentence syntax used for an image different from the person image; When the determination unit determines that the captured image is the person image, the first syntax sentence is output using characters having the predetermined meaning, and the captured image is an image different from the person image. And an output unit that outputs the sentence of the second syntax using the character having the predetermined meaning when it is determined by the determination unit.

  According to the present invention, more flexible character information can be given to a captured image.

It is an example of the functional block diagram of the image processing apparatus 1 by the 1st Embodiment of this invention. 4 is an example of a sentence template stored in a storage unit 90. 4 is an example of words stored in a storage unit 90. It is explanatory drawing for demonstrating extraction of the color scheme of a captured image. 3 is a flowchart illustrating an example of the operation of the image processing apparatus 1. 3 is a flowchart illustrating an example of the operation of the image processing apparatus 1. It is an example of the captured image with which the text was added by the text addition part. It is an example of the functional block diagram of the imaging device 100 by the 2nd Embodiment of this invention. It is a figure which shows typically an example of the process which extracts the feature-value of a captured image. It is a figure which shows typically another example of the process which extracts the feature-value of a captured image. It is a flowchart which shows typically the determination method of a smile level. It is a figure which shows an example of the output image from an image processing apparatus. It is a figure which shows another example of the output image from an image processing apparatus. It is a schematic block diagram showing the internal structure of the image process part of an imaging device. It is a flowchart which shows the flow of determination of a representative color. It is a conceptual diagram which shows an example of the process in an image process part. It is a conceptual diagram which shows an example of the process in an image process part. It is a conceptual diagram which shows the result of the clustering implemented with respect to the main area | region shown in FIG. It is an example of the image which the text was added by the text addition part. It is another example of the image which the text was added by the text addition part. It is a figure which shows an example of the correspondence table of a color and a word. It is a figure which shows an example of the correspondence table for a distant view image (2nd scene image). It is a figure which shows an example of the corresponding | compatible table for other images (3rd scene image).

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is an example of a functional block diagram of an image processing apparatus 1 according to the first embodiment of the present invention. FIG. 2 is an example of a sentence template stored in the storage unit 90. FIG. 3 is an example of words stored in the storage unit 90. FIG. 4 is an explanatory diagram for describing extraction of a color arrangement pattern of a captured image.

  As illustrated in FIG. 1, the image processing apparatus 1 includes an image input unit 10, a determination unit 20, a sentence creation unit 30, a sentence addition unit 40, and a storage unit 90. For example, the image input unit 10 inputs a captured image via a network or a storage medium. The image input unit 10 outputs the captured image to the determination unit 20.

  The storage unit 90 stores a sentence template that completes a sentence by inserting a word into a predetermined blank part. Specifically, in the storage unit 90, as a text template, a person image template used for creating a text for an image in which a person is a subject (hereinafter referred to as a person image), and a landscape (also referred to as a second type) are subjects. And a landscape image template used for creating a sentence for an image (hereinafter referred to as a landscape image). An example of a person image is a portrait (also referred to as a first type).

  For example, the storage unit 90 stores two types of person image templates as shown in FIGS. The person image templates shown in FIGS. 2 (a) and 2 (b) correspond to a blank portion (blank portion {number}) for inserting words corresponding to the number of subjects and a color arrangement pattern of the captured image. It has a blank space (blank space {adjective}) for inserting a word.

  Further, for example, the storage unit 90 stores two types of landscape image templates as shown in FIGS. Note that the landscape image template shown in FIG. 2C corresponds to a blank part (blank part {date}) for inserting a word corresponding to the imaging condition (date and time) of the captured image and a color arrangement pattern of the captured image. It has a blank space for inserting a word. In addition, the landscape image template shown in FIG. 2D includes a blank part (denoted as blank part {place}) for inserting a word corresponding to the imaging condition (location) of the captured image, and a color arrangement pattern of the captured image. It has a blank space for inserting a corresponding word.

  The person image template described above is a sentence template that is focused on a person imaged as a subject, that is, a sentence template in which a blank portion is set in a sentence from the viewpoint of a person imaged as a subject. is there. For example, the word “pause” of the person image template in FIG. 2A and the word “pose” of the person image template in FIG. 2B express the viewpoint of the imaged person. The landscape image template described above is a text template that is imaged from the entire captured image, that is, a text template in which a blank portion is set in text from the viewpoint of the photographer who captured the subject. For example, the word “scene” in the landscape image template in FIG. 2C and the word “scenery” in the landscape image template in FIG. 2D express the viewpoint of the photographer.

  Further, the storage unit 90 stores a word to be inserted into each blank portion of the sentence template in addition to the sentence template (person image template, landscape image template). For example, as illustrated in FIG. 3A, the storage unit 90 stores a word related to the number of people as a word to be inserted into the blank portion {number of people} in association with the number of subjects of the captured image.

  For example, when the person image template is used and the number of subjects is “1”, the word “one person” is inserted into the blank portion {number of persons] of the person image template. Note that the sentence creation unit 30 reads a sentence template to be used from the storage unit 90 and inserts a word into the blank part (described later).

  Further, as shown in FIG. 3B, the storage unit 90 is inserted into the blank portion {adjective} of the person image template or the blank portion {adjective} of the landscape image template in association with the color arrangement pattern of the captured image. The adjective for the person image and the adjective for the landscape image are stored as the words to be processed.

  For example, when a person image template is used, the color arrangement pattern of the entire region of the captured image is the first color “color 1”, the second color “color 2”, and the third color “ When the color is “3”, the word “cool” is inserted into the blank portion {adjective} of the person image template. Further, when the landscape image template is used, the color arrangement pattern of the entire area of the captured image has the first color “color 2”, the second color “color 1”, and the third color “ When the color is “4”, the word “lively” is inserted into the blank portion {adjective} of the landscape image template.

The above-mentioned colors 1 to 5 are obtained by classifying individual colors actually expressed in the captured image into five colors (five representative colors) based on a standard such as a warm color / cold color. In other words, the above-mentioned colors 1 to 5 are obtained by classifying the pixel values of each pixel of the captured image into five colors based on, for example, a warm color / cold color standard.
The first color constituting the color arrangement pattern is the color most expressed in the captured image among colors 1 to 5, and the second color is the second most frequently expressed in the captured image among colors 1 to 5. The expressed color, the third color, is the third most frequently expressed color in the captured image among the colors 1 to 5. In other words, the color having the largest number of pixels classified when the pixel values are classified into colors 1 to 5 is the first color, and the number of pixels classified when the pixel values are classified into colors 1 to 5 is The second most common color is the second color, and when the pixel values are classified into colors 1 to 5, the third most common color is the third color.
Note that the text creation unit 30 extracts a color arrangement pattern from the captured image.

Note that a color arrangement pattern in a partial area on the captured image may be used instead of the color arrangement pattern in the entire area of the captured image. That is, the sentence creation unit 30 may insert an adjective corresponding to the color arrangement pattern of a partial area on the captured image into the blank part. Specifically, the text creation unit 30 determines a predetermined area on the captured image according to whether the captured image is a person image or a landscape image, and according to the determined color arrangement pattern of the predetermined area on the captured image An adjective may be inserted in the blank.
For example, when the captured image is a person image as shown in FIG. 4C, the sentence creation unit 30 determines the central area of the human image as a predetermined area, and sets the color arrangement pattern of the central area. An adjective corresponding to the extracted color scheme may be inserted into the blank section. In addition, when the captured image is a landscape image as shown in FIG. 4D, the text creation unit 30 determines the upper region of the landscape image as a predetermined region, and extracts the color arrangement pattern of the region. Alternatively, an adjective corresponding to the extracted color arrangement pattern may be inserted into the blank section.

  Although not shown, the storage unit 90 associates with the imaging date and time, and inserts a word related to the date and time as a word to be inserted into the blank {date and time} (for example, time, “good morning”, “dusk”, “midsummer”). "!", ...) is memorized. In addition, the storage unit 90 associates the location-related words (for example, “Northern country”, “Old city”, “Mt. Fuji”, “Kaminarimon”,. Remember.

  The determination unit 20 acquires a captured image from the image input unit 10. The determination unit 20 determines whether the acquired captured image is a person image or a landscape image. Hereinafter, the determination of the person image / landscape image by the determination unit 20 will be described in detail. Note that the first threshold value (also referred to as “Flow”) is smaller than the second threshold value (also referred to as “Fhigh”).

The determination unit 20 attempts to recognize a face area in the captured image.
(When face area = 0)
When no face area is recognized in the captured image, the determination unit 20 determines that the captured image is a landscape image.

(When face area = 1)
When the determination unit 20 recognizes one face area in the captured image, the determination unit 20 calculates a ratio R of the size of the face area to the size of the captured image according to the following equation (1).
R = Sf / Sp (1)
Sp in the above formula (1) is the size of the captured image, and specifically, the length in the longitudinal direction of the captured image is used. Sf in the above formula (1) is the size of the face area. Specifically, the length in the longitudinal direction of the rectangle circumscribing the face area (or the length of the major axis of the ellipse surrounding the face area ( Long diameter)) is used.

  The determination unit 20 that has calculated the ratio R compares the ratio R with the first threshold value Flow. If the determination unit 20 determines that the ratio R is less than the first threshold value Flow, the determination unit 20 determines that the captured image is a landscape image. On the other hand, when the determination unit 20 determines that the ratio R is equal to or greater than the first threshold value Flow, the determination unit 20 compares the ratio R with the second threshold value Fhigh.

  If the determination unit 20 determines that the ratio R is equal to or greater than the second threshold Fhigh, the determination unit 20 determines that the captured image is a person image. On the other hand, when the determination unit 20 determines that the ratio R is less than the second threshold value Fhigh, the determination unit 20 determines that the captured image is a landscape image.

(If face area ≧ 2)
When the determination unit 20 recognizes a plurality of face regions in the captured image, the determination unit 20 calculates a ratio R (i) of the size of each face region to the size of the captured image according to the following equation (2).
R (i) = Sf (i) / Sp (2)
Sp in the above formula (2) is the same as that in the above formula (1). Sf (i) in the above formula (2) is the size of the i-th face area. Specifically, the length of the rectangle circumscribing the i-th face area (or the face area is The major axis length (major axis) of the enclosing ellipse is used.

  The determination unit 20 that has calculated R (i) calculates the maximum value (Rmax) of R (i). That is, the determination unit 20 calculates the ratio Rmax of the maximum face area size to the size of the captured image.

  The determination unit 20 that has calculated the ratio Rmax compares the ratio Rmax with the first threshold value Flow. If the determination unit 20 determines that the ratio Rmax is less than the first threshold value Flow, the determination unit 20 determines that the captured image is a landscape image. On the other hand, when the determination unit 20 determines that the ratio Rmax is greater than or equal to the first threshold value Flow, the determination unit 20 compares the ratio Rmax with the second threshold value Fhigh.

  If the determination unit 20 determines that the ratio Rmax is greater than or equal to the second threshold Fhigh, the determination unit 20 determines that the captured image is a person image. On the other hand, when the determination unit 20 determines that the ratio Rmax is less than the second threshold Fhigh, the determination unit 20 calculates the standard deviation σ of R (i). The following formula (3) is a formula for calculating the standard deviation σ.

  The determination unit 20 that has calculated the standard deviation σ compares the standard deviation σ with a third threshold value (also referred to as Fstdev). If the determination unit 20 determines that the standard deviation σ is less than the third threshold Fstdev, the determination unit 20 determines that the captured image is a person image. On the other hand, when the determination unit 20 determines that the standard deviation σ is equal to or greater than the third threshold Fstdev, the determination unit 20 determines that the captured image is a landscape image.

  As described above, when the determination unit 20 recognizes a plurality of face areas in the captured image, the ratio Rmax of the maximum face area to the size of the captured image is equal to or greater than the second threshold Fhigh. Determines that the captured image is a person image. In addition, even when the ratio Rmax is less than the second threshold Fhigh, the determination unit 20 determines that the standard deviation σ of the ratio R (i) of the plurality of face regions is not greater than the first threshold Flow. When it is less than the third threshold value Fstdev, it is determined that the captured image is a person image.

  Note that the determination unit 20 replaces the determination with the standard deviation σ of the ratio R (i) of the plurality of face areas and the third threshold value Fstdev, and the variance λ and the distribution λ of the ratio R (i) of the plurality of face areas The determination may be made using the threshold value. Further, the determination unit 20 may use the standard deviation (or variance) of the plurality of face regions Sf (i) instead of the standard deviation (or variance) of the ratio R (i) of the plurality of face regions ( In this case, a threshold value for the face area Sf (i) is used).

  If the determination unit 20 determines that the captured image is a human image, the determination unit 20 determines (counts) the number of subjects based on the number of face areas having a ratio R (i) equal to or greater than the first threshold value Flow. That is, the determination unit 20 determines each face area having a ratio R (i) equal to or greater than the first threshold value Flow as each subject, and determines the number of face areas equal to or greater than the first threshold value Flow as the number of subjects. To do.

The determination unit 20 outputs the determination result to the sentence creation unit 30. Specifically, when the determination unit 20 determines that the captured image is a person image, the image determination result information indicating the determination result indicating that the captured image is a person image, and the number determination result information indicating the determination result of the number of subjects. Is output to the sentence creation unit 30. On the other hand, when the determination unit 20 determines that the captured image is a landscape image, the determination unit 20 outputs image determination result information indicating a determination result indicating that the image is a landscape image to the sentence creation unit 30.
In addition, the determination unit 20 outputs the captured image acquired from the image input unit 10 to the sentence creation unit 30.

  The text creation unit 30 acquires a determination result and a captured image from the determination unit 20. The sentence creation unit 30 reads out either a person image template or a landscape image template from the storage unit 90 according to the acquired determination result. Specifically, when the text creation unit 30 acquires image determination result information indicating a determination result indicating that the image is a person image, the sentence creation unit 30 selects from two types of person image templates stored in the storage unit 90. One of the randomly selected person image templates is read out. In addition, when acquiring image determination result information indicating a determination result indicating that the image is a landscape image, the text creation unit 30 selects at random from two types of landscape image templates stored in the storage unit 90. The one person image template thus read is read out.

  The sentence creation unit 30 creates a sentence for the captured image by inserting a word corresponding to the feature amount or the imaging condition of the captured image into the blank part of the read sentence template (person image template or landscape image template). The word corresponding to the feature amount is an adjective corresponding to the color arrangement pattern of the captured image, or a word corresponding to the number of subjects (word related to the number of subjects). The word corresponding to the imaging condition of the captured image is a word corresponding to the imaging date and time (word related to the date and time) or a word corresponding to the imaging location (word related to the location).

  As an example, when the person image template shown in FIG. 2A is read, the text creation unit 30 acquires the number of subjects of the captured image from the number determination result information and stores it in association with the number of persons. The read word (word related to the number of people) is read from the storage unit 90 and inserted into the blank portion {number of people}, the color arrangement pattern of the captured image is extracted, and the word stored in association with the extracted color arrangement pattern (Adjective for person image) is read from the storage unit 90 and inserted into the blank part {adjective} to create a sentence for the captured image. Specifically, if the number of subjects is “1” and the color arrangement pattern is the first color “color 1”, the second color “color 2”, and the third color “color 3”, the sentence creation unit 30 Create the sentence "Cool memories spent alone".

  As another example, when the person image template shown in FIG. 2B is read out, the sentence creating unit 30 reads out words related to the number of people from the storage unit 90, as in FIG. 2A. Are inserted into the blank portion {number of people}, and the adjectives for the person image are read from the storage portion 90 and inserted into the blank portion {adjective} to create a sentence for the captured image. Specifically, if the number of subjects is “10” and the color arrangement pattern is the first color “color 5”, the second color “color 4”, and the third color “color 2”, the sentence creation unit 30 Sentence "Hot feeling? Pause with many people! ! Is created.

As another example, when the landscape image template shown in FIG. 2C is read, the text creation unit 30 acquires the imaging date and time from the additional information (for example, Exif) of the captured image, and acquires the acquired imaging date and time. The word stored in association with the word (word related to the date and time) is read from the storage unit 90 and inserted into the blank field {date and time}, the color arrangement pattern of the captured image is extracted, and is associated with the extracted color arrangement pattern. The stored word (adjective for landscape image) is read from the storage unit 90 and inserted into the blank field {adjective} to create a sentence for the captured image.
Specifically, when the word “Midsummer !!!” is stored in the storage unit 90 in association with August, the imaging date and time is August 10, 2011, the color arrangement pattern is the first color “color 5”, If the second color is “color 4” and the third color is “color 2”, the sentence creation unit 30 determines the sentence “Midsummer! ! . Create a piece that feels hot.

As another example, when the landscape image template shown in FIG. 2D is read, the text creation unit 30 acquires an imaging location from the additional information of the captured image and associates it with the acquired imaging location. The stored words (words related to the place) are read from the storage unit 90 and inserted into the blank portion {place}, the color arrangement pattern of the captured image is extracted, and stored in association with the extracted color arrangement pattern. A word (adjective for a landscape image) is read from the storage unit 90 and inserted into a blank portion {adjective} to create a sentence for the captured image.
Specifically, when the word “old city” is stored in the storage unit 90 in association with Kyoto Station, the imaging location is in front of Kyoto Station, the color arrangement pattern is the first color “Color 1”, and the second color “Color 2”. ”, The third color“ color 5 ”, the sentence creation unit 30 reads the sentence“ Old city. The soft scenery at that time! Is created.

  The sentence creation unit 30 that created the sentence outputs the created sentence and the captured image to the sentence addition unit 40. The text adding unit 40 acquires a text and a captured image from the text creating unit 30. The sentence adding unit 40 adds (synthesizes) the sentence to the captured image.

  Subsequently, the operation of the image processing apparatus 1 will be described. 5 and 6 are flowcharts showing an example of the operation of the image processing apparatus 1.

  In FIG. 5, the image input unit 10 inputs a captured image (step S10). The image input unit 10 outputs the captured image to the determination unit 20. The determination unit 20 determines whether or not there is one or more face areas in the captured image (step S12). When determining that there is one or more face areas in the captured image (step S12: Yes), the determination unit 20 calculates a ratio of the size of the face area to the size of the captured image for each face area (step S14). ), And calculates the maximum value of the ratio (step S16).

  Subsequent to step S16, the determination unit 20 determines whether or not the maximum value calculated in step S16 is greater than or equal to the first threshold (step S20). If the determination unit 20 determines that the maximum value calculated in step S16 is greater than or equal to the first threshold (step S20: Yes), the determination unit 20 determines whether the maximum value is greater than or equal to the second threshold ( Step S22). If the determination unit 20 determines that the maximum value is equal to or greater than the second threshold (step S22: Yes), the determination unit 20 determines that the captured image is a person image (step S30). Subsequent to step S30, the determination unit 20 counts the number of face areas having a ratio equal to or higher than the first threshold as the number of subjects (step 32). Subsequent to step S32, the determination unit 20 writes a determination result (image determination result information indicating a determination result indicating that the image is a person image and number determination result information indicating a determination result of the number of subjects) and the captured image as text. The data is output to the creation unit 30.

  On the other hand, when it is determined in step S22 that the maximum value is less than the second threshold (step S22: No), the determination unit 20 determines whether there are two or more face regions in the captured image ( Step S40). When determining that there are two or more face regions in the captured image (step S40: Yes), the determination unit 20 calculates the standard deviation of the ratio calculated in step S14 (step S42), and the standard deviation is It is determined whether or not the threshold value is less than 3 (step S44). If the determination unit 20 determines that the standard deviation is less than the third threshold (step S44: Yes), the process proceeds to step S30.

  On the other hand, when it is determined in step S12 that there is no face area in the captured image (step S12: No), or when it is determined in step S20 that the maximum value is less than the first threshold (step S20). : No), or when it is determined in step S40 that there is only one face area in the captured image (step S40: No), the determination unit 20 determines that the captured image is a landscape image (step S50). ). Subsequent to step S <b> 50, the determination unit 20 outputs a determination result (image determination result information indicating a determination result indicating that the image is a landscape image) to the sentence creation unit 30.

  The above-described step S40 is a process for preventing a captured image with one face area from being always determined to be a person image. In step S40 described above, there are a very large number of very small face areas with the same size in addition to the face area having the largest ratio of the size of the face area to the size of the captured image. If so, the standard deviation is small, so that it may be determined that the image is a person image. Therefore, in order to reduce the above-described determination as much as possible, the determination unit 20 may determine whether there are two or more face regions having a predetermined size. For example, the determination unit 20 may determine whether there are two or more face regions in which the above-described ratio is equal to or greater than a first threshold value.

  Subsequent to step S32 or step S50, the sentence creation unit 30 reads out either a person image template or a landscape image template from the storage unit 90 according to the determination result acquired from the determination unit 20, A word corresponding to the feature amount or the imaging condition of the captured image is inserted into the blank portion of the read sentence template to create a sentence for the captured image (step S100).

  FIG. 6 shows details of step S100. In FIG. 6, the text creation unit 30 determines whether or not the captured image is a person image (step S102). Specifically, if the sentence creation unit 30 has acquired image determination result information indicating a determination result indicating that the image is a person image as a determination result from the determination unit 20, the captured image is a person image. If image determination result information indicating a determination result indicating that the image is a landscape image has been acquired, it is determined that the captured image is not a person image.

  When the sentence creation unit 30 determines that the captured image is a person image (step S102: Yes), the sentence creation unit 30 reads a person image template from the storage unit 90 (step S104). Specifically, the sentence creation unit 30 reads one person image template randomly selected from the two types of person image templates stored in the storage unit 90.

  Subsequent to step S104, the sentence creation unit 30 inserts a word corresponding to the number of subjects in the blank portion {number of people} of the person image template (step S110). Specifically, the sentence creation unit 30 acquires the number of subjects from the number determination result information, reads words stored in association with the number of persons (words related to the number of persons) from the storage unit 90, and reads the person image. Insert it into the blank field {number of people} of the template.

  Subsequent to step S110, the sentence creation unit 30 inserts a word corresponding to the color arrangement pattern of the captured image (person image) into a blank portion {adjective} of the person image template (step S120). Specifically, the sentence creation unit 30 extracts a color arrangement pattern in the central area of the captured image (person image), and stores a word (adjective for person image) stored in association with the color arrangement pattern. The data is read from the part 90 and inserted into the blank part {adjective} of the person image template.

  On the other hand, when the text creation unit 30 determines in step S102 that the captured image is a landscape image (step S102: No), the text creation unit 30 reads a landscape image template from the storage unit 90 (step S106). Specifically, the text creation unit 30 reads one landscape image template selected at random from two types of landscape image templates stored in the storage unit 90.

  Subsequent to step S106, the sentence creation unit 30 inserts a word corresponding to the color arrangement pattern of the captured image (landscape image) into the blank portion {adjective} of the landscape image template (step S130). Specifically, the text creation unit 30 extracts a color arrangement pattern of the upper region of the captured image (landscape image), and stores a word (landscape image adjective) stored in association with the color arrangement pattern. The data is read from 90 and inserted into the blank field {adjective} of the landscape image template.

  Subsequent to step S120 or step S130, the sentence creation unit 30 determines whether or not a blank part {date} exists in the read sentence template (step S132). In the case of the present embodiment, as shown in FIG. 2, the landscape image template of FIG. 2C has a blank portion {date and time}, but the person image template of FIGS. The landscape image template in FIG. 2D does not have a blank field {date and time}. Accordingly, when the landscape image template shown in FIG. 2C is read out in step S106, the text creation unit 30 determines that the blank part {date} exists, and in step S104, the sentence creation unit 30 determines in FIG. Alternatively, when the person image template of FIG. 2B is read out, or when the landscape image template of FIG. 2D is read out in step S106, the blank portion {date} does not exist. to decide.

  When the sentence creation unit 30 determines that the blank part {date} exists in the read sentence template (step S132: Yes), the word corresponding to the imaging condition (date) of the captured image is changed to a blank part {date} of the sentence template. (Step S140). Specifically, the text creation unit 30 acquires the imaging date and time from the additional information of the captured image (landscape image), and stores the words (words related to the date and time) stored in association with the imaging date and time. Is inserted into the blank field {date and time} of the landscape image template. On the other hand, if the text creation unit 30 determines that the blank text {date} does not exist in the read text template (step S132: No), the process skips step S140 and proceeds to step S142.

  Subsequent to step S132 (No) or step S140, the sentence creation unit 30 determines whether or not a blank part {place} exists in the read sentence template (step S142). In the case of the present embodiment, as shown in FIG. 2, the landscape image template in FIG. 2D has a blank portion {place}, but the person image template in FIGS. The landscape image template in FIG. 2C does not have a blank portion {place}. Accordingly, when the landscape image template shown in FIG. 2D is read out in step S106, the text creation unit 30 determines that a blank portion {place} exists, and in step S104, the sentence creation unit 30 determines that the blank image portion {location} exists. Alternatively, when the person image template of FIG. 2B is read out, or when the landscape image template of FIG. 2C is read out in step S106, the blank portion {place} does not exist. to decide.

  When the sentence creation unit 30 determines that the blank part {place} exists in the read sentence template (step S142: Yes), the word corresponding to the imaging condition (place) of the captured image is changed to a blank part {place} of the sentence template. (Step S150). Specifically, the text creation unit 30 acquires an imaging location from the additional information of the captured image (landscape image), and stores a word (word related to the location) stored in association with the imaging location. Is inserted into the blank field {place} of the landscape image template. Then, the flowchart shown in FIG. 6 ends, and the process returns to the flowchart shown in FIG. On the other hand, when the sentence creation unit 30 determines that the blank part {place} does not exist in the read sentence template (step S142: No), step S150 is skipped and the process returns to the flowchart shown in FIG.

  Returning to FIG. 5, the sentence creation unit 30 that created the sentence outputs the created sentence and the captured image to the sentence addition unit 40. The text adding unit 40 acquires a text and a captured image from the text creating unit 30. The text adding unit 40 adds (synthesizes) the text acquired from the text creating unit 30 to the captured image acquired from the text creating unit 30. Then, the flowchart shown in FIG. 5 ends.

  FIG. 7 is an example of a captured image to which a sentence is added by the sentence adding unit 40. The captured image in FIG. 7A is determined to be a person image because one person's face is greatly reflected. That is, it is determined that the maximum value of the ratio of the size of the face area to the size of the captured image (the ratio of the one face area) is greater than or equal to the second threshold (Step S22 (Yes )). The captured image in FIG. 7B is determined to be a person image because the faces of the two people are large. That is, it is determined that the maximum value of the ratio of the size of the face area to the size of the captured image is greater than or equal to the second threshold (step S22 (Yes)).

  The captured image in FIG. 7C is determined to be a person image because a face of a certain size is captured and the size is uniform. That is, in the captured image, the maximum value of the ratio of the size of the face area to the size of the captured image is greater than or equal to the first threshold and less than the second threshold (step S22 (No)), but the standard deviation is larger. It is determined that it is less than the third threshold (step S44 (Yes)).

  The captured image in FIG. 7D shows a face of a certain size, but the size is not uniform, so it is determined to be a landscape image. That is, in the captured image, the maximum value of the ratio of the size of the face area to the size of the captured image is greater than or equal to the first threshold and less than the second threshold (step S22 (No)), but the standard deviation is larger. It is determined that it is greater than or equal to the third threshold (step S44 (No)). The captured image in FIG. 7E is determined to be a landscape image because no face is captured (step S12 (No)).

  As described above, according to the image processing apparatus 1, more flexible character information can be given to the captured image. In other words, the image processing apparatus 1 classifies captured images into human images and landscape images, and for human images, creates a human image text using a human image template stored in advance, For landscape images, landscape image text is created using a prestored landscape image template, so that more flexible text information can be given according to the captured content.

  In the above-described embodiment, the image input unit 10 has described an example in which the captured image is output to the determination unit 20 when the captured image is input. However, the manner in which the determination unit 20 acquires the captured image is not limited thereto. For example, the image input unit 10 may store the captured image in the storage unit 90 when the captured image is input, and the determination unit 20 may read and acquire a desired captured image from the storage unit 90 when necessary.

  In the above-described embodiment, the example in which the number of colors of the first color constituting the color arrangement pattern uses five colors of colors 1 to 5 is described. However, for convenience of explanation, the number of colors may be six or more. Good. The same applies to the second color and the third color. In the above-described embodiment, an example using a color arrangement pattern including three colors of the first color to the third color has been described, but the number of colors constituting the color arrangement pattern is not limited to this. For example, a color arrangement pattern composed of two colors or four or more colors may be used.

  In the above embodiment, when the captured image is a person image, the sentence creation unit 30 reads one of the person image templates randomly selected from the two types stored in the storage unit 90. However, the mode of selecting one of the two types of person image templates to be read is not limited to this. For example, the text creation unit 30 may select one person image template designated by the user via the operation unit (not shown). Similarly, the text creation unit 30 may select one landscape image template designated by the user via the designation receiving unit.

  In the above-described embodiment, an example has been described in which a word to be inserted into the blank portion of the selected template is always obtained from the storage unit 90. However, a word to be inserted into the blank portion of the selected template is obtained from the storage unit 90. If not, another template may be selected again. For example, when the landscape image template of FIG. 2D having a blank space {location} is selected for creating a sentence of a captured image, but the imaging location cannot be acquired from the additional information of the captured image. The landscape image template may be selected again in FIG. 2C without the blank portion {place}.

  In the above embodiment, the image processing apparatus 1 has described the example in which the person image template having the blank part {number of people} and the blank part {adjective} is stored in the storage unit 90. The number and type of parts are not limited to this. For example, the person image template may have one or both of the blank part {date} and the blank part {location} in addition to the blank part {number of people} and the blank part {adjective}. Further, when the image processing apparatus 1 includes various sensors, the human image template includes a blank portion {blank portion {illuminance}) for inserting a word corresponding to the imaging condition (illuminance) of the captured image, and an imaging condition of the captured image ( It may have a blank part {blank part {temperature}) for inserting a word corresponding to (temperature).

Further, the person image template does not necessarily have a blank portion {number of people}. An example of the case where the person image template does not have a blank portion {number of people} is a case where a sentence including words corresponding to the number of subjects is not created for the person image. When a sentence including words corresponding to the number of subjects is not created for a person image, the image processing apparatus 1 naturally needs to store a person image template having a blank portion {number of persons] in the storage unit 90. There is no.
Another example of the case where the person image template does not have the blank portion {number of people} is a case where a plurality of person image templates corresponding to the number of subjects are stored in the storage unit 90. When storing a plurality of person image templates corresponding to the number of subjects in the storage unit 90, the image processing apparatus 1 inserts a word corresponding to the number of subjects in the blank portion {number} for the person image. Thus, instead of creating a sentence including words according to the number of subjects, a person image template according to the number of subjects is read from the storage unit 90 and a sentence including words according to the number of subjects is created. .

  In the above embodiment, the image processing apparatus 1 also includes a landscape image template having a blank portion {date} and a blank portion {adjective}, and a landscape image template having a blank portion {location} and a blank portion {adjective}. However, the number and type of blank sections included in the landscape image template are not limited to this. For example, when the image processing apparatus 1 includes various sensors, the blank section {illuminance}, the blank section {temperature}, and the like may be included.

  In the above embodiment, the image processing apparatus 1 has described an example in which two types of person image templates are stored in the storage unit 90. However, one type or three or more types of person image templates are stored in the storage unit 90. May be. Similarly, the image processing apparatus 1 may store one type or three or more types of landscape image templates in the storage unit 90.

  In the above-described embodiment, the image processing apparatus 1 has been described as an example in which the sentence is added to the captured image when the sentence for the captured image is created. The image may be stored in the storage unit 90 in association with the captured image.

  The storage unit 90 also has a first syntax that is a syntax of a sentence used for an image of a first type (for example, a portrait) and a syntax of a sentence that is used for an image of a second type (for example, a landscape). Two syntaxes may be stored.

  When the first syntax and the second syntax are stored in the storage unit 90 and the determination unit 20 determines that the captured image is the first type image (that is, the determination unit 30) When it is determined that 20 is a human image), a sentence of the first syntax is created using a predetermined text, and when the determination unit 20 determines that the captured image is a second type image (that is, When the determination unit 20 determines that the image is a landscape image), a sentence having the second syntax may be created using a predetermined text.

  The image processing apparatus 1 further includes a determination unit (not shown) that determines text corresponding to at least one of the feature amount of the captured image and the imaging condition (text according to the feature amount of the captured image or / and the imaging condition). You may do it. For example, when the image input unit 10 inputs (acquires) a captured image, the determination unit determines a text corresponding to a feature amount or / and an imaging condition of the captured image as predetermined text used for document creation. More specifically, for example, a plurality of texts are stored in advance in the storage unit 90 in association with the feature amount and the imaging condition, and the determination unit selects the feature amount or / or from the plurality of texts in the storage unit 90. The text corresponding to the imaging condition is selected.

  That is, when the determination unit 20 determines that the captured image is the first type image, the sentence creation unit 30 creates a sentence of the first syntax using the text determined by the determination unit as described above. When the determination unit 20 determines that the captured image is the second type of image, the determination unit generates a sentence having the second syntax using the text determined as described above.

(Second Embodiment)
Subsequently, a second embodiment of the present invention will be described with reference to the drawings. FIG. 8 is an example of a functional block diagram of the imaging apparatus 100 according to the second embodiment of the present invention.

As illustrated in FIG. 8, the imaging apparatus 100 includes an imaging unit 110, a buffer memory unit 130, an image processing unit 140, a display unit 150, a storage unit 160, a communication unit 170, an operation unit 180, and a CPU (Central processing unit) 190. Prepare.
Note that the image processing unit 140 of the imaging apparatus 100 corresponds to the determination unit 20, the sentence creation unit 30, and the sentence addition unit 40 of the image processing apparatus 1 according to the first embodiment. The storage unit 160 of the imaging apparatus 100 corresponds to the storage unit 90 of the image processing apparatus 1 according to the first embodiment.

  The imaging unit 110 includes an optical system 111, an imaging element 119, and an A / D conversion unit 120. The optical system 111 includes one or more lenses.

  For example, the image sensor 119 converts an optical image formed on the light receiving surface into an electrical signal and outputs the electrical signal to the A / D converter 120.

  Further, the image sensor 119 uses the image data obtained when an imaging instruction is received via the operation unit 180 as captured image data of a captured still image via the A / D conversion unit 120 and the image processing unit 140. And stored in the storage medium 200.

  In addition, for example, the imaging element 119 uses the A / D conversion unit 120 and the image processing as the through image data (captured image) as image data obtained continuously in a state where an imaging instruction is not received via the operation unit 180. The data is continuously output to the display unit 150 via the unit 140.

  Note that the optical system 111 may be attached to and integrated with the imaging apparatus 100, or may be detachably attached to the imaging apparatus 100.

  The A / D converter 120 performs analog / digital conversion on the electronic signal converted by the image sensor 119, and outputs captured image data (captured image) that is the converted digital signal.

  That is, the imaging unit 110 is controlled by the CPU 190 based on the set imaging conditions (for example, aperture value, exposure value, etc.), and forms an optical image via the optical system 111 on the imaging device 119 for A / D conversion. A captured image based on the optical image converted into a digital signal by the unit 120 is generated.

  The operation unit 180 includes, for example, a power switch, a shutter button, a cross key, a confirmation button, and other operation keys. The operation unit 180 is operated by the user, receives a user operation input, and outputs it to the CPU 190.

  The image processing unit 140 performs image processing on the image data stored in the buffer memory unit 130 based on the image processing conditions stored in the storage unit 160. For example, the image processing unit 140 executes the processes of the determination unit 20, the text creation unit 30, and the text addition unit 40 of the image processing apparatus 1 according to the first embodiment. The image data stored in the buffer memory unit 130 is image data input to the image processing unit 140. For example, the image data read from the captured image data, the through image data, or the storage medium 200 described above. This is taken image data.

  The display unit 150 is a liquid crystal display, for example, and displays image data, an operation screen, and the like. For example, the display unit 150 displays a captured image to which text is added by the image processing unit 140.

  The storage unit 160 stores various information. Specifically, the storage unit 160 stores at least information stored in the storage unit 90 of the image processing apparatus 1 according to the first embodiment.

  The buffer memory unit 130 temporarily stores image data captured by the imaging unit 110. The communication unit 170 is connected to a removable storage medium 200 such as a card memory, and writes, reads, or deletes photographed image data in the storage medium 200.

  The storage medium 200 is a storage unit that is detachably connected to the imaging device 100, and stores, for example, photographed image data generated by the imaging unit 110. The CPU 190 controls each component included in the imaging device 100. The bus 300 is connected to the imaging unit 110, the CPU 190, the operation unit 180, the image processing unit 140, the display unit 150, the storage unit 160, the buffer memory unit 130, and the communication unit 170, and outputs from each unit. The transferred image data and control signals are transferred.

  A program for executing each process of the image processing apparatus 1 according to the first embodiment is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed. By doing so, the above-described various processes related to each process of the image processing apparatus 1 may be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  FIG. 9 is a diagram schematically illustrating an example of a process for extracting a feature amount of a captured image used for determining a sentence arranged on an image. In the example of FIG. 9, the determination unit of the image processing apparatus classifies the scene of the captured image into a person image or a landscape image. Next, the image processing apparatus extracts a feature amount of the captured image according to the scene. The feature amount can be the number of faces (number of subjects) and the average color (color arrangement pattern) in the case of a person image, and can be the average color (color arrangement pattern) in the case of a landscape image. . Based on these feature quantities, words (adjectives and the like) to be inserted into the person image template or the landscape image template are determined.

  Here, in the example of FIG. 9, the color arrangement pattern is configured by a combination of a plurality of representative colors constituting the captured image. Therefore, the color arrangement pattern can represent the average color (average color) of the captured image. In one example, “first color”, “second color”, and “third color” are defined as a color arrangement pattern, and based on a combination of these three colors, that is, based on three average colors, a person image Or a word (adjective) to be inserted into a text template for a landscape image.

  In the example of FIG. 9, the scene of the captured image is classified into two types (person image and landscape image). In another example, the scene of the captured image can be classified into three or more types (3, 4, 5, 6, 7, 8, 9, or 10 types or more).

  FIG. 10 is a diagram schematically illustrating another example of the process of extracting the feature amount of the captured image used for determining the text arranged on the image. In the example of FIG. 10, the scene of the captured image can be classified into three or more types.

  In the example of FIG. 10, the determination unit of the image processing apparatus determines whether the captured image is a person image (first mode image), a distant view image (second mode image), or another image (third mode image). judge. First, as in the example of FIG. 9, the determination unit determines whether the captured image is a person image or an image different from the person image.

  Next, when the captured image is an image different from the human image, the determination unit determines whether the captured image is a distant view image (second mode image) or another image (third mode image). To do. This determination can be performed using, for example, a part of the image identification information given to the captured image.

  Specifically, in order to determine whether the captured image is a distant view image, a focal length that is a part of the image identification information can be used. The determination unit determines that the captured image is a distant view image when the focal distance is greater than or equal to a preset reference distance, and determines the captured image as another image when the focal distance is less than the reference distance. As described above, the captured image is classified into three types of scenes: a person image (first mode image), a distant view image (second mode image), or another image (third mode image). Note that examples of distant view images (second mode images) include landscape images such as the sea and mountains, and examples of other images (third mode images) include flowers and pets.

  Also in the example of FIG. 10, after the scene of the captured image is classified, the image processing apparatus extracts the feature amount of the captured image in accordance with the scene.

  In the example of FIG. 10, when the captured image is a person image (first scene image), the number of faces (number of subjects) and the feature amount of the captured image used for determining the text arranged on the image are A smile level can be used. That is, when the captured image is a human image, a word to be inserted into the human image template may be determined based on the determination result of the smile level in addition to or instead of the determination result of the number of faces (number of subjects). it can. Hereinafter, an example of a smile level determination method will be described with reference to FIG.

  In the example of FIG. 11, the determination unit of the image processing apparatus detects a face area from a person image by a method such as face recognition (step S5001). In one example, the degree of smile of a person image is calculated by digitizing the degree of ascending of the mouth corner. For example, various known techniques for face recognition can be used for calculating the smile level.

  Next, the determination unit compares the smile level with the first smile threshold α set in advance (step S5002). When it is determined that the smile level is greater than or equal to α, the determination unit determines that the smile level of the person image is “smile: large”.

  On the other hand, when it is determined that the smile level is less than α, the determination unit compares the smile level with a second smile threshold value β set in advance (step S5003). When it is determined that the smile level is β or more, the determination unit determines that the smile level of this person image is “smile: medium”. Furthermore, when it is determined that the smile level is less than β, the determination unit determines that the smile level of the person image is “smile: small”.

  A word to be inserted into the person image template is determined based on the determination result of the smile level of the person image. Here, examples of the word corresponding to the smile level of “smile: large” include “full of joy” and “very good”. Examples of words that correspond to the smile level of “smile: medium” include “joyful” and “good calm”. Examples of words corresponding to the smile level of “smile: small” include “seriously seems” and “cool”.

  In the above description, the case where the word inserted into the person image template is a continuous form has been described. However, the present invention is not limited to this, and may be an end form, for example. In this case, examples of words corresponding to the smile level of “smile: large” include “smile is nice”, “it is a very good smile”, and the like. Examples of words corresponding to the smile level of “smile: medium” include “smiley” and “good expression”. Examples of words corresponding to a smile level of “smile: small” include “looks serious” and “looks serious”.

  FIG. 12A is an example of an output image showing the operation result of the image processing apparatus, and this output image has a sentence determined based on the example of FIG. In the example of FIG. 12A, the captured image is determined to be a person image, and the number of subjects and the color arrangement pattern (average color) are extracted as the feature amount. Further, the word inserted into the person image template is determined as “heavy” according to the color arrangement pattern. As a result, the output result shown in FIG. 12A is obtained. That is, in the example of FIG. 12A, the word “heavy” (adjective, combined form) is determined based on the average color of the captured image.

  FIG. 12B is another example of an output image showing the operation result of the image processing apparatus, and this output image has a sentence determined based on the example of FIG. In the example of FIG. 12B, the captured image is determined to be a person image, and the number of subjects and the smile level are extracted as the feature amount. Further, according to the smile level, the word inserted into the person image template is determined as “good expression”. As a result, the output result shown in FIG. 12B is obtained. That is, in the example of FIG. 12B, the word (end form) of “good expression” is determined based on the smile level of the person in the captured image. Like the output result of FIG. 12B, by using the word output using the smile level for the person image, it is possible to attach character information that is relatively close to the impression received from the image.

  Returning to FIG. 10, when the captured image is a landscape image (second scene image) or another image (third scene image), as a feature amount of the captured image used to determine the text arranged on the image, A representative color can be used instead of the average color. As the representative color, the “first color” in the color arrangement pattern, that is, the most frequently used color in the captured image can be used. Alternatively, the representative color can be determined using clustering as described below.

  FIG. 13 is a schematic block diagram illustrating an internal configuration of an image processing unit included in the imaging apparatus. In the example of FIG. 13, the image processing unit 5040 of the image processing apparatus includes an image data input unit 5042, an analysis unit 5044, a text creation unit 5052, and a text addition unit 5054. The image processing unit 5040 performs various types of analysis processing on the image data generated by the imaging unit or the like, thereby acquiring various types of information regarding the content of the image data, and creating text that is highly consistent with the content of the image data. Then, text can be added to the image data.

  The analysis unit 5044 includes a color information extraction unit 5046, a region extraction unit 5048, and a clustering unit 5050, and performs analysis processing on the image data. The color information extraction unit 5046 extracts first information regarding color information of each pixel included in the image data from the image data. Typically, the first information is a total of the HSV values of all the pixels included in the image data. However, the first information is the frequency at which the predetermined color appears in the image (frequency in pixel units, area ratio, etc.) for a predetermined color associated with similarity (for example, associated with a predetermined color space). The color resolution and the type of color space are not limited.

  For example, the first information may be information indicating how many pixels of each color are included in the image data for each color represented by an HSV space vector (HSV value) or RGB value. . However, the color resolution in the first information may be changed as appropriate in consideration of the burden of calculation processing, and the type of color space is not limited to HSV or RGB, and may be CMY, CMYK, or the like.

  FIG. 14 is a flowchart showing the flow of representative color determination performed in the analysis unit 5044. In step S5101, the image processing apparatus starts calculating the representative color of specific image data 5060 (captured image, see FIG. 15).

  In step S5102, the image data input unit 5042 of the image processing apparatus outputs the image data to the analysis unit 5044. Next, the color information extraction unit 5046 of the analysis unit 5044 calculates first information 5062 regarding the color information of each pixel included in the image data (see FIG. 15).

  FIG. 15 is a conceptual diagram illustrating a calculation process of the first information 5062 performed by the color information extraction unit 5046 in step S5102. The color information extraction unit 5046 aggregates the color information included in the image data 5060 for each color (for example, for each gradation of 256 gradations) to obtain first information 5062. The histogram shown in the lower part of FIG. 15 represents an image of the first information 5062 calculated by the color information extraction unit 5046. The horizontal axis of the histogram in FIG. 15 is color, and the vertical axis represents how many pixels of a predetermined color are included in the image data 5060.

  In step S5103 of FIG. 14, the region extraction unit 5048 of the analysis unit 5044 extracts the main region in the image data 5060. For example, the area extraction unit 5048 extracts a focused area from the image data 5060 shown in FIG. 15, and recognizes the central portion of the image data 5060 as the main area (see the main area 5064 in FIG. 16). ).

  In step S5104 in FIG. 14, the region extraction unit 5048 of the analysis unit 5044 determines a clustering target region to be implemented in step S5105. For example, as shown in the upper part of FIG. 16, the area extraction unit 5048 recognizes that part of the image data 5060 is the main area 5064 in step S5103 and extracts the main area 5064, the clustering target is set as the main area 5064. The first information 5062 (main first information 5066) corresponding to the area 5064 is used. The histogram shown in the lower part of FIG. 16 represents an image of the main first information 5066.

  On the other hand, if the region extracting unit 5048 has not extracted the main region 5064 in the image data 5060 in step S5103, the region extracting unit 5048 first information corresponding to all the regions of the image data 5060 as shown in FIG. 5062 is determined as a clustering target. Note that there is no difference in the subsequent processing between the case where the main region 5064 is extracted and the case where it is not extracted, except that the target region for clustering is different. I will explain.

  In step S5105 of FIG. 14, the clustering unit 5050 of the analysis unit 5044 performs clustering on the main first information 5066 that is the first information 5062 of the area determined in step S5104. FIG. 17 is a conceptual diagram showing the result of clustering performed by the clustering unit 5050 on the main first information 5066 of the main region 5064 shown in FIG.

  For example, the clustering unit 5050 classifies the main first information 5066 (see FIG. 16) having 256 gradations into a plurality of clusters by the k-means method. Note that the clustering is not limited to the k-means method (k average method). In other examples, other methods such as the shortest distance method can be used.

  The upper part of FIG. 17 shows to which cluster each pixel is classified, and the histogram shown in the lower part of FIG. 17 shows the number of pixels belonging to each cluster. By the clustering by the clustering unit 5050, the 256 first main information 5066 (FIG. 16) is classified into less than 256 clusters (three in the example shown in FIG. 17). The result of clustering can include information about the size of each cluster and information about the color of each cluster (the position of the cluster in the color space).

  In step S5106, the clustering unit 5050 of the analysis unit 5044 determines a representative color of the image data 5060 based on the clustering result. In one example, when the clustering unit 5050 obtains a clustering result as illustrated in FIG. 17, the color belonging to the maximum cluster 5074 including the most pixels among the plurality of calculated clusters is set as the representative color of the image data 5060. To do.

  When the calculation of the representative color is completed, the sentence creation unit 5052 creates a text using information on the representative color and assigns the text to the image data 5060.

  The text creation unit 5052 reads a text template for a landscape image, for example, and applies a word (for example, “2012/03/10”) corresponding to the generation date and time of the image data 5060 to {date and time} of the text template. In this case, the analysis unit 5044 can retrieve information related to the generation date and time of the image data 5060 from the storage medium and output the information to the text creation unit 5052.

  In addition, the sentence creation unit 5052 applies a word corresponding to the representative color of the image data 5060 to the {adjective} of the sentence template. The sentence creation unit 5052 reads the correspondence information from the storage unit 5028 and applies it to the sentence template. In one example, the storage unit 5028 stores a table in which colors and words are associated with each scene. The sentence creation unit 5052 can create a sentence (for example, “I found a very beautiful thing”) using words read from the table.

  FIG. 18 shows image data 5080 to which text is given by the series of processes described above.

  FIG. 19 shows an example of image data to which text is given by a series of processes similar to the above when the scene is a distant view image. In this case, the scene is classified as a distant view image, and the representative color is determined to be blue. For example, in a table in which colors and words are associated with each scene, the word “fresh” is associated with the representative color “blue”.

  FIG. 20 is a diagram illustrating an example of a table having correspondence information between colors and words. In the table of FIG. 20, a color and a word are associated with each scene of a person image (first scene image), a distant view image (second scene image), and another image (third scene image). In one example, when the representative color of the image data is “blue” and the scene is another image (third scene image), the sentence creation unit 5052 uses a word corresponding to the representative color (for example, from the correspondence information in the table). “Classy”) and select {adjective} in the sentence template.

  The correspondence table between colors and words can be set based on a color chart such as a PCCS color system, CICC color system, or NCS color system.

  FIG. 21 shows an example of a correspondence table for a distant view image (second scene image) using a color chart of the CCIC display system. FIG. 22 shows an example of a correspondence table for other images (third scene images) using a CCIC display color chart.

  In FIG. 21, the horizontal axis corresponds to the hue of the representative color, and the vertical axis corresponds to the tone of the representative color. By using the table of FIG. 21 to determine the word, not only the information on the hue of the representative color but also the information on the tone of the representative color is used to determine the word, and a text that is relatively close to the sensibility generated by humans is given. Is possible. Hereinafter, a specific text setting example in the case of a distant view image (second scene image) using the table of FIG. 21 will be described. In the case of other images (third scene images), the same setting can be made using the table of FIG.

  In FIG. 21, when it is determined that the representative color is the area A5001, the name of the representative color (red, orange, yellow, blue, etc.) is applied to the word in the text as it is. For example, if the hue of the representative color is “red (R)” and the tone is “Vivid Tone (V)”, the adjective “crimson” representing the color is selected.

  When the representative color is determined to be the color of the region A5002, A5003, A5004, or A5005, an adjective associated with the color is applied to the word in the text. For example, when the representative color is determined to be the color (green) of the area A5003, the adjectives associated with green, such as “comfortable” and “fresh”, are applied.

  When the representative color is determined to be the color of the area A5001 to A5005 and the tone is a vivid tone (V), a strong tone (S), a bright tone (B), or a pale tone (LT). Applies adverbs that indicate the degree before the adjectives (eg, very, pretty, etc.).

  When the representative color is determined to be the area A5006, that is, “white tone (white)”, “clean”, “clear”, and the like, which are words associated with white, are selected. Further, if the representative color is determined to be the area A5007, that is, a gray color (light gray tone: ltGY, medium gray tone: mGY, or dark gray tone: dkGY), it is a safe adjective. “Clean”, “nice”, etc. are selected. In an image in which a white or gray color, that is, an achromatic color is a representative color, various colors are often included in the entire image. Therefore, by using a word that is less related to color, it is possible to prevent a text having an inappropriate meaning from being added, and it is possible to provide a text that is relatively close to an image received from an image.

  Further, when the representative color does not belong to any of the areas A5001 to A5007, that is, when the representative color is a low tone (dark grayish tone) or black (black tone), characters having a predetermined meaning (Word or sentence) can be selected as text. Characters having a predetermined meaning include, for example, “where is here”, “a”, and the like. These words and sentences can be stored in the storage unit of the image processing apparatus as a “tweet dictionary”.

  In other words, when the representative color is determined to be low tone or black, it may be difficult to determine the hue of the entire image. Even in such a case, characters having less relation to the color are used as described above. Thus, it is possible to prevent a text having an inappropriate meaning from being added, and to add a text close to an image received from an image.

  In the above example, the case where the sentence and the word are uniquely determined according to the scene and the representative color has been described. However, the present invention is not limited to this, and exception processing is sometimes performed in the selection of the sentence and the word. it can. For example, the text may be extracted from the “tweet dictionary” once every plural times (for example, once every 10 times). As a result, the display content of the text is not necessarily patterned, so that the user can be prevented from getting bored with the display content.

  In the above example, the case where the sentence adding unit arranges the text generated by the sentence creating unit at the upper part or the lower part of the image has been described. However, the present invention is not limited to this. It can also be arranged.

  In the above example, the case where the position of the text is fixed in the image has been described. However, the present invention is not limited to this. For example, the text can be displayed so as to flow on the display unit of the image processing apparatus. Thereby, the input image is not easily affected by the text, or the text visibility is improved.

  In the above example, the case where the text is always pasted on the image has been described. However, the present invention is not limited to this. For example, in the case of a person image, the text is not pasted, and in the case of a distant view image or other images. You may make it paste a text.

  In the above example, the sentence adding unit has described the case where the display method (font, color, display position, etc.) of the text generated by the sentence creating unit is determined by a predetermined method. A variety of text display methods can be determined. Hereinafter, some examples of these methods will be described.

  In one example, the user can correct the text display method (font, color, display position) via the operation unit of the image processing apparatus. Alternatively, the user can change or delete the contents (words) of the text. In addition, the user can select not to display the entire text, that is, display / non-display of the text.

  In one example, the size of the text can be changed according to the scene of the input image. For example, when the scene of the input image is a person image, the text can be reduced, and when the scene of the input image is a distant view image or other images, the text can be increased.

  In one example, text can be highlighted and combined with image data. For example, when the input image is a person image, a balloon can be given to the person and text can be placed in the balloon.

  In one example, the display color of the text can be set with reference to the representative color of the input image. Specifically, a color having the same hue as the representative color of the input image and a different tone can be used as a text display color. As a result, it is possible to give a text that is in harmony with the input image without excessively claiming the text.

  In particular, when the representative color of the input image is white, exception processing may be performed in determining the text display color. Here, in the exception processing, for example, the text color can be set to white and the peripheral portion of the text can be set to black.

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus 10 ... Image input part 20 ... Determination part 30 ... Text preparation part 40 ... Text addition part 90 ... Memory | storage part 100 ... Imaging device 110 ... Imaging part 111 ... Optical system 119 ... Imaging element 120 AD conversion part 130 ... Buffer memory unit 140 ... Image processing unit 150 ... Display unit 160 ... Storage unit 170 ... Communication unit 180 ... Operation unit 190 ... CPU 200 ... Storage medium 300 ... Bus

Claims (11)

An image input unit for inputting a captured image;
As a sentence template for completing a sentence by inserting a word into a predetermined blank space, a person image template used for creating a sentence for a person image with a person as a subject and a sentence for a landscape image with a landscape as a subject A storage unit for storing a landscape image template to be used;
A determination unit that determines whether the captured image is the person image or the landscape image;
According to a determination result by the determination unit for the captured image, the sentence template of either the person image template or the landscape image template is read from the storage unit, and the blank part of the read sentence template is stored in the blank part. An image processing apparatus comprising: a sentence creation unit that inserts a word corresponding to a feature amount or an imaging condition of a captured image and creates a sentence for the captured image. The image processing apparatus according to claim 1.
The storage unit
Storing the sentence template in which the blank portion is set in a sentence from the viewpoint of a person imaged as a subject as the person image template;
An image processing apparatus, wherein the sentence template in which the blank portion is set in a sentence from a viewpoint of a photographer who has photographed a subject is stored as the landscape image template. The image processing apparatus according to claim 1 or 2,
The determination unit
In the person image, the number of subjects is further determined as the feature amount,
The sentence creation unit
An image processing apparatus, wherein a sentence is created by inserting a word corresponding to the number of subjects in the blank portion with respect to the person image. The image processing apparatus according to claim 3.
The determination unit
In the case where a plurality of face areas are recognized in the captured image,
The ratio of the size of the maximum face area to the size of the captured image is greater than or equal to a first threshold, less than a second threshold that is greater than or equal to the first threshold, and a plurality of face areas When the standard deviation or variance of the ratio or the standard deviation or variance of the sizes of the plurality of face regions is less than the third threshold,
Or, when the ratio of the size of the maximum face area is equal to or greater than the second threshold value,
An image processing apparatus characterized in that the captured image is determined to be the person image, and the number of subjects is determined based on the number of face areas having a ratio equal to or greater than the first threshold. The image processing apparatus according to any one of claims 1 to 4,
The sentence creation unit
An image processing apparatus, wherein a sentence is created by inserting an adjective corresponding to a color arrangement pattern of the captured image into the blank portion as a word corresponding to a feature amount of the captured image. The image processing apparatus according to claim 5.
The sentence creation unit
A sentence is created by inserting an adjective in accordance with a color arrangement pattern of a predetermined area on the captured image determined according to whether the captured image is the person image or the landscape image into the blank section. An image processing apparatus. An image input unit for inputting a captured image;
A determination unit that determines text corresponding to at least one of the feature amount of the captured image and the imaging condition of the captured image;
A determination unit that determines whether the captured image is a first type image or a second type image different from the first type;
A storage unit for storing a first syntax that is a syntax of a sentence used for the first type and a second syntax that is a syntax of a sentence used for the second type;
When the determination unit determines that the captured image is the first type image, the sentence having the first syntax is created using the text determined by the determination unit, and the captured image is the second image. An image processing apparatus, comprising: a sentence creation unit that creates a sentence of the second syntax using the text determined by the determination unit when the determination unit determines that the image is a type image. An image processing apparatus according to claim 7,
The image processing apparatus according to claim 1, wherein the first type is a portrait, and the second type is a landscape. An imaging unit that images a subject and generates a captured image;
As a sentence template for completing a sentence by inserting a word into a predetermined blank space, a person image template used for creating a sentence for a person image with a person as a subject and a sentence for a landscape image with a landscape as a subject A storage unit for storing a landscape image template to be used;
A determination unit that determines whether the captured image is the person image or the landscape image;
According to a determination result by the determination unit for the captured image, the sentence template of either the person image template or the landscape image template is read from the storage unit, and the blank part of the read sentence template is stored in the blank part. An imaging apparatus comprising: a sentence creation unit that inserts a word according to a feature amount or an imaging condition of a captured image and creates a sentence for the captured image. As a sentence template for completing a sentence by inserting a word into a predetermined blank space, a person image template used for creating a sentence for a person image with a person as a subject and a sentence for a landscape image with a landscape as a subject In a computer of an image processing apparatus provided with a storage unit for storing a landscape image template to be used,
An image input step for inputting a captured image;
A determination step of determining whether the captured image is the person image or the landscape image;
According to the determination result of the determination step for the captured image, the sentence template of either the person image template or the landscape image template is read from the storage unit, and the blank section of the read sentence template is stored in the blank section. A program for executing a sentence creating step of creating a sentence for a captured image by inserting a word corresponding to a feature amount or an imaging condition of the captured image. A determining unit that determines a character having a predetermined meaning from the captured image;
A determination unit that determines whether the captured image is a person image or an image different from the person image;
A storage unit that stores a first syntax that is a syntax of a sentence used for the person image and a second syntax that is a syntax of a sentence used for an image different from the person image;
When the determination unit determines that the captured image is the person image, the first syntax sentence is output using the characters having the predetermined meaning, and the captured image is different from the person image. And an output unit that outputs the sentence of the second syntax using the character having the predetermined meaning when it is determined by the determination unit.

Images