JP2013080464A - Image processing device, imaging device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, an imaging device, and a program which can reduce a load of calculation processing for labeling an image.SOLUTION: An image processing device comprises: an image acquisition unit for acquiring taken image data; a scene determination unit for determining a scene from the acquired image data; a main color extraction unit for extracting a main color on the basis of frequency distribution of color information from the acquired image data; a storage unit for prestoring color information and a first label in association for each scene; and a first label generation unit for reading out the first label prestored in association with the extracted main color and the determined scene from the storage unit, and generating the read first label as a label of the acquired image data.

Description

  The present invention relates to an image processing device, an imaging device, and a program.

In a conventional image processing apparatus that classifies images, an image is divided into predetermined pattern areas, and a histogram of distribution relating to the color of each area is created. In the conventional image processing apparatus, the most frequently appearing color exceeding a specific threshold is determined as the representative region color of the region. Further, in the conventional image processing apparatus, the feature amount of the region is extracted, and based on the determined feature amount of the region and the representative color, the image from which the feature amount is extracted is defined, and the image dictionary is constructed.
In a conventional image processing apparatus, for example, a representative color of a large area at the top of an image is extracted, and based on the extracted representative color, an image dictionary is defined by defining “blue sky”, “cloudy sky”, “night sky”, and the like. It comprised (for example, refer patent document 1).

Japanese Patent Laid-Open No. 2001-160057

  However, in the prior art described in Patent Document 1, since the classification is performed based on the feature amount extracted for each predetermined region and the representative color that is the most frequently appearing color, the calculation process for classifying (labeling) the image The burden of was great.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an imaging apparatus, and a program that can reduce the load of calculation processing for labeling an image.

  To achieve the above object, an image processing apparatus according to the present invention includes an image acquisition unit that acquires captured image data, a scene determination unit that determines a scene from the acquired image data, and the acquired image. A main color extracting unit that extracts a main color from data based on a frequency distribution of color information; a storage unit in which color information and a first label are associated in advance for each scene; and the extraction from the storage unit A first label generation unit that reads the first label stored in advance in association with the determined main color and the determined scene, and generates the read first label as a label of the acquired image data; It is characterized by providing.

  In addition, an imaging apparatus according to the present invention includes the image processing apparatus described above.

  Further, the present invention is a program for causing a computer to execute image processing of an image processing apparatus having an imaging unit, and an image acquisition procedure for acquiring captured image data, and a scene is determined from the acquired image data. A scene determination procedure, a main color extraction procedure for extracting a main color from the acquired image data based on a frequency distribution of color information, the extracted main color, and color information and a first label associated with each scene A first label generation procedure for reading the first label from a storage unit stored in advance and generating the read first label as a label of the acquired image data. It is said.

  An image processing apparatus according to an aspect of the present invention includes a scene determination unit that determines whether or not a person-captured scene, and a color from the image data when the scene determination unit determines that the scene is not a person-captured scene. A color extracting unit that extracts information; a storage unit that stores color information and characters having a predetermined meaning in association with each other; and the scene determination unit that determines that the color is not a person-captured scene. And a reading unit that reads out the character having the predetermined meaning corresponding to the color information extracted by the extraction unit from the storage unit.

  According to the image processing apparatus of the present invention, suitable image labeling can be realized. In addition, according to the present invention, it is possible to provide an imaging apparatus and a program including the image processing apparatus.

It is a schematic block diagram which shows the structure of the imaging system 1 which concerns on this embodiment. 2 is a block diagram of an image processing unit 4 according to the embodiment. FIG. It is a figure explaining an example of the image identification information memorize | stored linked | related with the image data in the storage medium 200 based on the embodiment. It is a figure explaining an example of the combination of the main color memorize | stored in the table memory | storage part 45 based on the embodiment, and a 1st label. It is a figure explaining an example of the main colors of the image data concerning the embodiment. It is a figure explaining an example of labeling of the main color extracted in FIG. It is a figure of the image data and color vector of the sports which concern on the embodiment. It is a figure of portrait image data and color vectors according to the embodiment. It is a figure of the image data and color vector of the landscape concerning the embodiment. It is a figure explaining an example of the 1st label of the combination of main colors for every scene concerning the embodiment. It is a figure explaining the example of the 1st label by time, a season, and a color vector concerning the embodiment. 6 is a flowchart of label generation performed by the imaging apparatus 100 according to the embodiment. It is a block diagram of the image processing part 4a which concerns on 2nd Embodiment. It is a block diagram of the image processing part 4b which concerns on 3rd Embodiment. 6 is a flowchart of label generation performed by the imaging apparatus 100 according to the embodiment. It is a figure explaining an example which extracts a several color vector from the image data which concerns on 4th Embodiment. It is a figure which shows typically an example of the process which extracts the feature-value of a captured image. It is a figure which shows typically another example of the process which extracts the feature-value of a captured image. It is a flowchart which shows typically the determination method of a smile level. It is a figure which shows an example of the output image from an image processing apparatus. It is a figure which shows another example of the output image from an image processing apparatus. It is a schematic block diagram showing the internal structure of the image process part of an imaging device. It is a flowchart which shows the flow of determination of a representative color. It is a conceptual diagram which shows an example of the process in an image process part. It is a conceptual diagram which shows an example of the process in an image process part. It is a conceptual diagram which shows the result of the clustering implemented with respect to the main area | region shown in FIG. It is an example of the image which the text was added by the text addition part. It is another example of the image which the text was added by the text addition part. It is a figure which shows an example of the correspondence table of a color and a word. It is a figure which shows an example of the correspondence table for a distant view image (2nd scene image). It is a figure which shows an example of the corresponding | compatible table for other images (3rd scene image).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a schematic block diagram illustrating a configuration of an imaging system 1 according to the present embodiment.
An imaging apparatus 100 illustrated in FIG. 1 includes an imaging unit 2, a camera control unit 3, an image processing unit 4, a storage unit 5, a buffer memory unit 6, a display unit 7, an operation unit 11, a communication unit 12, a power supply unit 13, and a bus. 15 is provided.

The imaging unit 2 includes a lens unit 21, an imaging element 22, and an AD conversion unit 23. The imaging unit 2 images a subject and generates image data. The imaging unit 2 is controlled by the camera control unit 3 based on the set imaging conditions (for example, aperture value, exposure, etc.), and the imaging device 22 captures the optical image of the subject input through the lens unit 21. Form an image on the surface. In addition, the imaging unit 2 converts the analog signal output from the imaging element 22 into a digital signal in the AD conversion unit 23 to generate image data.
Note that the lens unit 21 described above may be attached to and integrated with the imaging apparatus 100, or may be detachably attached to the imaging apparatus 100.

  The imaging element 22 outputs an analog signal obtained by photoelectrically converting an optical image formed on the imaging surface to the AD conversion unit 23. The AD converter 23 converts the analog signal input from the image sensor 22 into a digital signal, and outputs image data that is the converted digital signal.

  For example, the imaging unit 2 outputs image data of a captured still image in response to a still image shooting operation on the operation unit 11. Further, the imaging unit 2 outputs image data of moving images continuously captured at a predetermined interval in accordance with a moving image shooting operation in the operation unit 11. Then, still image data and moving image data captured by the imaging unit 2 are recorded in the storage medium 200 via the buffer memory unit 6 and the image processing unit 4 under the control of the camera control unit 3. The imaging unit 2 outputs image data obtained continuously at a predetermined interval as through image data (through image) in a shooting standby state in which no shooting operation is performed in the operation unit 11. The through image data obtained by the imaging unit 2 is displayed on the display unit 7 via the buffer memory unit 6 and the image processing unit 4 under the control of the camera control unit 3.

  The image processing unit 4 performs image processing on the image data stored in the buffer memory unit 6 based on the image processing conditions stored in the storage unit 5. Here, the image data stored in the buffer memory unit 6 or the storage medium 200 is, for example, still image data, through image data, or moving image data captured by the imaging unit 2, or the storage medium 200. The image data read out from.

The storage unit 5 stores predetermined shooting conditions, image processing conditions, reproduction control conditions, display control conditions, recording control conditions, output control conditions, and the like for controlling the imaging apparatus 100. For example, the storage unit 5 is a ROM (Read Only Memory).
The storage unit 5 may record captured image data of moving images and image data of still images. In this case, for example, the storage unit 5 may be a flash memory or the like.

  The buffer memory unit 6 is used as a work area when the camera control unit 3 controls the imaging apparatus 100. The still image image data, the through image data, or the moving image data captured by the imaging unit 2 or the image data read from the storage medium 200 is buffered in the course of image processing under the control of the camera control unit 3. It is temporarily stored in the unit 6. The buffer memory unit 6 is, for example, a RAM (Random Access Memory).

  The display unit 7 is, for example, a liquid crystal display, and is an image based on the image data captured by the imaging unit 2 or an image based on the image data read from the storage medium 200, a menu screen, or the operation of the imaging device 100. Displays information about status and settings.

  The operation unit 11 includes an operation switch for an operator to input an operation to the imaging apparatus 100. For example, the operation unit 11 includes a power switch, a release switch, a mode switch, a menu switch, an up / down / left / right selection switch, a confirmation switch, a cancel switch, and other operation switches. Each of the above-described switches provided in the operation unit 11 outputs an operation signal corresponding to each operation to the camera control unit 3 in response to the operation.

A removable storage medium 200 such as a card memory is inserted into the communication unit 12.
Writing, reading, or erasing of image data is executed on the storage medium 200 via the communication unit 12.
The storage medium 200 is a storage unit that is detachably connected to the imaging apparatus 100, and stores, for example, image data generated by being captured by the imaging unit 2. In the present embodiment, the image data recorded in the storage medium 200 is, for example, a file in an Exif (Exchangeable Image File Format) format.

  The power supply unit 13 supplies power to each unit included in the imaging device 100. The power supply unit 13 includes, for example, a battery, and converts the voltage of power supplied from the battery into the operating voltage in each of the above-described units. Then, the power supply unit 13 supplies the converted power of the operating voltage to the above-described units under the control of the camera control unit 3 based on the operation mode (for example, the shooting operation mode or the sleep mode) of the imaging device 100.

  The bus 15 is connected to the imaging unit 2, the camera control unit 3, the image processing unit 4, the storage unit 5, the buffer memory unit 6, the display unit 7, the operation unit 11, and the communication unit 12, and image data output from each unit. And transfer control signals.

  The camera control unit 3 controls each unit included in the imaging device 100.

FIG. 2 is a block diagram of the image processing unit 4 according to the present embodiment.
As shown in FIG. 2, the image processing unit 4 includes an image acquisition unit 41, an image identification information acquisition unit 42 (scene determination unit), a color space vector generation unit 43, a main color extraction unit 44, a table storage unit 45, and a first storage unit 45. A label generation unit 46, a second label generation unit 47, and a label output unit 48 are provided.

  The image acquisition unit 41 reads the image data captured by the imaging unit 2 and the image identification information stored in association with the image data from the storage medium 200 via the bus 15. The image data read by the image acquisition unit 41 is image data selected by the user of the imaging system 1 by operating the operation unit 11. The image acquisition unit 41 outputs the acquired image data to the color space vector generation unit 43. The image acquisition unit 41 outputs the acquired image identification information to the image identification information acquisition unit 42.

FIG. 3 is a diagram illustrating an example of image identification information stored in association with image data in the storage medium 200 according to the present embodiment.
In FIG. 3, the left column is an example of items, and the right column is an example of information. As shown in FIG. 3, the items stored in association with the image data are the imaging date and time, the resolution of the entire image, the shutter speed, the aperture value (F value), the ISO sensitivity, the metering mode, the presence / absence of flash use, the scene Mode, still image / moving image, etc. These pieces of image identification information are information set by the photographer using the operation unit 11 of the imaging system 1 at the time of imaging, and information automatically set by the imaging apparatus 100. Further, the Exif standard information stored in association with the image data may be used as the image identification information.
In the item, “scene” (also referred to as a shooting mode) is a combination pattern such as shutter speed, F value, ISO sensitivity, and focal length set in advance in the imaging apparatus 100. These combination patterns are preset according to the object to be imaged, stored in the storage medium 200, and manually selected by the user from the operation unit 11. The scene is, for example, portrait, landscape, sport, night view portrait, party, beach, snow, sunset, night view, close-up, cooking, museum, fireworks, backlight, children, pets, and the like.

  Returning to FIG. 2, the image identification information acquisition unit 42 extracts the shooting information set in the captured image data from the image identification information output by the image acquisition unit 41, and uses the extracted shooting information as the first label. The data is output to the generation unit 46. Note that the shooting information is information necessary for the first label generation unit 46 to generate the first label, such as a scene and shooting date / time.

The color space vector generation unit 43 converts the image data output from the image acquisition unit 41 into a vector of a predetermined color space. The predetermined color space is, for example, HSV (Hue (Hue), Saturation (Saturation), Brightness (Brightness))).
The color space vector generation unit 43 classifies all pixels of the image data for each color vector, detects the frequency for each color vector, and generates a color vector frequency distribution. The color space vector generation unit 43 outputs information indicating the frequency distribution of the generated color vector to the main color extraction unit 44.
When the image data is HSV, the color vector is expressed as the following equation (1).

  In equation (1), i, j, and k are natural numbers of 0 to 100, respectively, when the hue is normalized to 0 to 100%.

The main color extraction unit 44 extracts three colors as main colors in order of frequency from the information indicating the frequency distribution of the color vectors output from the color space vector generation unit 43, and generates the first label information indicating the extracted main colors. To the unit 46. In addition, a color with high frequency is a color with many pixels of the same color vector. The information indicating the main color is the color vector of Expression (1) and the frequency (number of pixels) for each color vector.
In the present embodiment, the main color extraction unit 44 may be configured by the color space vector generation unit 43 and the main color extraction unit 44.

  In the table storage unit 45 (storage unit), a first label is stored in association with each scene and each combination of main colors.

FIG. 4 is a diagram illustrating an example of combinations of primary colors and first labels stored in the table storage unit 45 according to the present embodiment.
As shown in FIG. 4, among the main colors extracted from the image data, the first color having the highest frequency, the second color having the second highest frequency after the first color, and the second frequency having the second highest frequency. A first label is defined in advance for each combination of the three colors of the third color and for each scene, and is stored in the table storage unit 45. For example, when the first color is color 1, the second color is color 2, and the third color is color 3, the first label of scene 1 is label (1,1) and the label of scene n is label (1 , N). Similarly, in the combination where the first color is the color m, the second color is the color m, and the third color is the color m, the first label of the scene 1 is the label (m, 1), and the label of the scene n is the label ( m, n).
In this way, labels for each scene and each combination of the three main colors are defined in advance through experiments, questionnaires, etc., and stored in the table storage unit 45. The frequency ratio of the first color, the second color, and the third color is 1: 1: 1.

  Returning to FIG. 2, the first label generation unit 46 stores the first information stored in association with the shooting information output from the image identification information acquisition unit 42 and the information indicating the main color output from the main color extraction unit 44. The label is read from the table storage unit 45. The first label generation unit 46 outputs information indicating the read first label and information indicating the main color output by the main color extraction unit 44 to the second label generation unit 47. In addition, the first label generation unit 46 determines a scene using, for example, information included in Exif that is shooting information.

  The second label generation unit 47 extracts the frequency for each color vector from the information indicating the main color output from the main color extraction unit 44, normalizes the frequency of the three color vectors using the extracted frequency, and outputs the three main colors. Calculate the color ratio. The second label generation unit 47 generates a modification label (third label) for modifying the first label based on the calculated ratio of the three main colors, and the first label generation unit 46 outputs the generated modification label. The first label is modified by modifying it to one label to generate a second label for the image data. The second label generation unit 47 outputs information indicating the generated second label to the label output unit 48.

  The label output unit 48 stores information indicating the second label output by the second label generation unit 47 in the table storage unit 45 in association with the image data. Alternatively, the label output unit 48 stores information indicating the label output by the second label generation unit 47 in the storage medium 200 in association with the image data.

FIG. 5 is a diagram illustrating an example of main colors of image data according to the present embodiment.
In FIG. 5, the horizontal axis is a color vector, and the vertical axis is the frequency of a color vector (color information).
The example of FIG. 5, the color space vector generation unit 43, the color decomposed image data into HSV vector _{(HSV = (i m, j} m, k m); m is a natural number from 0 to 100) a graph of the frequency distribution of It is. FIG. 5 schematically shows H (hue) = 0, S (saturation) = 0, V (lightness) = 0 on the left end, and H = 100, S = 100, V = 100 on the right end in order. It is a thing. The result of calculating the frequency for each color vector is schematically shown. In the example of FIG. 5, the first color c1 having the highest frequency is the vector HSV = (i ₁ , j ₆₉ , k ₁₀₀ ) and rose (rose). The second color c2 having the second highest frequency after the first color is a vector HSV = (i ₁₃ , j ₅₂ , k ₁₀₀ ) and light yellow (sulfur). Further, the third color c3 having the second highest frequency after the second color is a vector HSV = (i ₄₀ , j ₆₅ , k ₈₀ ), and a bitumen magnetic color (emerald).

FIG. 6 is a diagram for explaining an example of labeling of main colors extracted in FIG. The color vectors in FIGS. 5 and 6 will be described assuming that the scene mode is portrait image data, for example.
FIG. 6A is an example of the first color, the second color, and the third color extracted in FIG. As shown in FIG. 6A, the color vectors are schematically arranged from the left in the order of the color vectors shown in FIG. The first label generation unit 46 reads the first label stored in association with the combination of the first color, the second color, and the third color extracted by the main color extraction unit 44 from the table storage unit 45. In this case, the first label of the combination of the first color, the second color, and the third color is stored as “fun”. Further, as shown in FIG. 6A, the widths of the first color, the second color, and the third color before normalization are L1, L2, and L3, and the lengths of the widths L1, L2, and L3 are equal. The length L10 is the sum of the widths L1, L2, and L3.

  In FIG. 6B, the extracted first color, second color, and third color are normalized with frequency, and the widths of the first color, second color, and third color are expressed as L1 ′, L2 ′, It is a figure after correct | amending like L3 '. The total width L10 is the same as that in FIG. In the example of FIG. 6B, since the frequency of the first color is higher than the frequencies of the other second and third colors, the second label generating unit 47 reads the first label read by the first label generating unit 46. For “pleasant”, based on a predetermined rule, a decoration label “very” that modifies the first label “fun” is generated. The predetermined rule is that when the first color is more frequent than the predetermined threshold value than the other second and third colors, the second label generating unit 47 determines that the modified label “ The first label is modified by generating the “very” and modifying the generated modified label to the first label “fun” to generate the second label “very fun”. The modification label is, for example, a word that emphasizes the first label.

Next, examples of modification labels will be described.
As shown in FIG. 6A, before normalization, the width or area of the three colors extracted by the main color extraction unit 44 is 1: 1: 1. Then, after normalization based on the frequency of the color vector, the widths or areas of the three colors are corrected as shown in FIG. For example, when the ratio of the first color is larger than about 67% of the entire L10, the second label generation unit 47 modifies the first label by modifying the first label by modifying “very” as the modification label. 2 labels. When the ratio of the first color is greater than about 50% and smaller than 67% of the entire L10, the second label generation unit 47 determines that there is no decoration label. That is, the second label generation unit 47 sets the first label as the second label without correcting it. When the ratio of the first color is about 33% of the entire L10, the second label generation unit 47 modifies the first label by modifying “first” as the decoration label to “second”. Label.
As described above, the second label generation unit 47 generates a modified label to be modified according to the first label. For example, for each first label, a modifiable modification label may be stored in association with the table storage unit 45 in advance.

Next, examples of main colors for each scene will be described with reference to FIGS.
FIG. 7 is a diagram of sports image data and color vectors according to the present embodiment. FIG. 7A shows sports image data, and FIG. 7B is a graph of sports color vectors. FIG. 8 is a diagram of portrait image data and color vectors according to the present embodiment. FIG. 8A shows portrait image data, and FIG. 8B is a graph of portrait color vectors. FIG. 9 is a diagram of landscape image data and color vectors according to the present embodiment. FIG. 9A is a landscape image data, and FIG. 9B is a graph of a landscape color vector. In FIG. 7B, FIG. 8B, and FIG. 9B, the horizontal axis is the color vector, and the vertical axis is the frequency (number of pixels).

  As shown in FIGS. 7A and 7B, the image data in FIG. 7A is decomposed into color vectors for each pixel, and the frequency (number of pixels) of each color vector is graphed. )become that way. The main color extraction unit 44 extracts, for example, three colors c11, c12, and c13 having a large number of pixels from such color vector information.

As shown in FIGS. 8A and 8B, the image data of FIG. 8A is decomposed into color vectors for each pixel, and the frequency (number of pixels) of each color vector is graphed. )become that way. The main color extraction unit 44 extracts, for example, three colors c21, c22, and c23 having a large number of pixels from such color vector information.
As shown in FIGS. 9A and 9B, the image data of FIG. 9A is decomposed into color vectors for each pixel, and the frequency (number of pixels) of each color vector is graphed. )become that way. The main color extraction unit 44 extracts, for example, three colors c31, c32, and c33 having a large number of pixels from such color vector information.

FIG. 10 is a diagram illustrating an example of a first label of a combination of main colors for each scene according to the present embodiment. In FIG. 10, rows represent scenes and columns represent color vectors.
In FIG. 10, when the image data is HSV, the hue, saturation, and intensity of the HSV of the combination of colors (color 1, color 2, color 3) are, for example, color 1 (94, 100, 25) (maroon) , Maroon), color 2 is (8, 100, 47) (tobacco color, coffee brown), and color 3 is (81, 100, 28) (deep purple, Dusky Violet).
Further, the hue, saturation, and intensity of the HSV of the color vector (color 4, color 5, and color 6) are, for example, that the color 4 is (1, 69, 100) (rose, rose) and the color 5 is (13, 25, 100) (ivory color, ivory) and color 6 is (52, 36, 91) (light blue, aqua blue).
Further, the hue, saturation, and intensity of the HSV of the color vector (color 7, color 8, color 9) are, for example, that color 7 is (40, 65, 80) (dark blue magnetic color, emerald) and color 8 is ( 0, 0, 100) (white, white) and color 9 is (59, 38, 87) (salvia, salvia blue).

As shown in FIG. 10, for example, when the combination of colors is (color 1, color 2, color 3), the first label whose portrait is the scene is stored in the table storage unit 45 as “dandy”. . The first label in which the scene is a landscape with the same color combination (color 1, color 2, and color 3) is stored in the table storage unit 45 as being “interesting”. Further, the first label in which the scene is sports even in the same color combination (color 1, color 2, and color 3) is stored in the table storage unit 45 as “(Rugby style) masculine”.
Also, as shown in FIG. 10, for example, when the color combination is (color 4, color 5, color 6), the first label whose scene is portrait is “childish” in the table storage unit 45. It is remembered. The first label in which the scene is a landscape with the same color combination (color 4, color 5, and color 6) is stored in the table storage unit 45 as “soft”. Further, the first label in which the scene is a sport even in the same color combination (color 4, color 5, and color 6) is stored in the table storage unit 45 as “(tennis style) lively”.
Also, as shown in FIG. 10, for example, when the combination of colors is (color 7, color 8, color 9), the table storage unit 45 indicates that the first label whose scene is portrait is “youthful”. Is remembered. The first label in which the scene is a landscape with the same color combination (color 7, color 8, and color 9) is stored in the table storage unit 45 as “(fresh green image) refreshing”.
In addition, the first label in which the scene is sports even in the same color combination (color 7, color 8, and color 9) is stored in the table storage unit 45 as “(sea sports style) refreshing”.
In addition, as shown in FIG. 10, the information stored in the table storage unit 45 stores not only the color combination and the first label of the adjective and adverb but also the word representing the image in association with each other. Also good. Note that the word representing an image is, for example, a rugby image or a fresh green image.

FIG. 11 is a diagram illustrating an example of a first label based on time, season, and color vector according to the present embodiment. In FIG. 11, the color vector is HSV image data, and is the combination of colors described in FIG. 10 (color 7, color 8, and color 9). In FIG. 11, the columns represent time and season, and the rows are labels for each time and season for color combinations (color 7, color 8, color 9).
As shown in FIG. 11, the first label of the color combination (color 7, color 8, color 9) is “fresh” when the time is morning, “rainy” when the time is noon, In the case of night, it is stored in the table storage unit 45 that “dawn is near”.
As shown in FIG. 11, the first label of the color combination (color 7, color 8, color 9) is “chilly” when the season is spring, “cool” when the season is summer, and “cool” when the season is autumn When the season is winter, the table storage unit 45 stores “cold”.
For such information regarding the time and season, the first label generation unit 46 reads the first label from the table storage unit 45 based on the shooting date and time included in the image identification information acquired by the image identification information acquisition unit 42.
Moreover, as shown in FIG. 11, the first label may be the same in spring and autumn for the same color combination (color 7, color 8, color 9).

  Next, label generation processing performed by the imaging apparatus 100 will be described with reference to FIG. FIG. 12 is a flowchart of label generation performed by the imaging apparatus 100 according to the present embodiment.

(Step S <b> 1) The imaging unit 2 of the imaging apparatus 100 captures an image based on the control of the camera control unit 3. Next, the imaging unit 2 converts the captured image data into digital data by the AD conversion unit 23, and stores the converted image data in the storage medium 200.
Next, the camera control unit 3 captures image identification information including imaging conditions set or selected by the user by the operation unit 11 during imaging and information automatically set or acquired by the imaging apparatus 100 during imaging. The image data is stored in the storage medium 200 in association with the image data. After step S1, the process proceeds to step S2.

(Step S2) Next, the image acquisition unit 41 of the image processing unit 4 stores the image data captured by the imaging unit 2 via the bus 15 and the image identification information stored in association with the image data. Read from the medium 200. The image data read by the image acquisition unit 41 is image data selected by the user of the imaging system 1 by operating the operation unit 11.
Next, the image acquisition unit 41 outputs the acquired image data to the color space vector generation unit 43. Next, the image acquisition unit 41 outputs the acquired image identification information to the image identification information acquisition unit 42. After step S2, the process proceeds to step S3.

  (Step S3) Next, the image identification information acquisition unit 42 extracts the shooting information set in the image data captured from the image identification information output by the image acquisition unit 41, and uses the extracted shooting information as the first label. The data is output to the generation unit 46. After step S3 ends, the process proceeds to step S4.

  (Step S4) Next, the color space vector generation unit 43 converts the image data output from the image acquisition unit 41 into a vector of a predetermined color space. The predetermined color space is, for example, HSV. Next, the color space vector generation unit 43 classifies all pixels of the image data for each generated color vector, detects the frequency for each color vector, and generates a color vector frequency distribution. Next, the color space vector generation unit 43 outputs information indicating the frequency distribution of the generated color vector to the main color extraction unit 44. After step S4 ends, the process proceeds to step S5.

  (Step S5) Next, the main color extraction unit 44 extracts three colors as main colors in order of frequency from the information indicating the frequency distribution of the color vectors output by the color space vector generation unit 43, and the extracted main colors are extracted. The indicated information is output to the first label generation unit 46. After step S5 ends, the process proceeds to step S6.

(Step S <b> 6) Next, the first label generation unit 46 stores the first label generation unit 46 associated with the shooting information output by the image identification information acquisition unit 42 and the information indicating the main color output by the main color extraction unit 44. One label is read from the table storage unit 45. Next, the first label generation unit 46 outputs information indicating the read first label and information indicating the main color output by the main color extraction unit 44 to the second label generation unit 47.
The first label generation unit 46 is stored in the table storage unit 45 in association with shooting information output from the image identification information acquisition unit 42 and information indicating the main color output from the main color extraction unit 44. If the first label is not stored, for example, it is determined whether or not the first label of another scene is recorded for the same main color. When it is determined that the first label of another scene is recorded for the same main color, the first label generation unit 46 reads the first label of another scene for the same main color from the table storage unit 45. Also good. On the other hand, when it is determined that the first label of another scene is not recorded for the same main color, the first label generation unit 46 associates the color with the same color vector that is the closest to the main color and the color vector. The stored label may be read from the table storage unit 45.
After step S6 ends, the process proceeds to step S7.

  (Step S7) Next, the second label generation unit 47 normalizes the frequency of each color vector from the information indicating the main color output from the main color extraction unit 44, and calculates the ratio of the three main colors. After step S7 ends, the process proceeds to step S8.

  (Step S8) Next, the second label generation unit 47 generates a modification label for modifying the first label output by the first label generation unit 46 based on the calculated ratio of the three main colors, and the generated modification The first label is modified by modifying the first label with the label to generate a second label. Next, the second label generation unit 47 outputs information indicating the generated second label to the label output unit 48. After step S8 ends, the process proceeds to step S9.

(Step S9) Next, the label output unit 48 stores information indicating the second label output from the second label generation unit 47 in the table storage unit 45 in association with the image data.
In step S6, when the first label stored in association with the information indicating the scene and the information indicating the main color is not stored in the table storage unit 45, the label output unit 48 determines in step S6. The read first label and the extracted main color may be associated with each other and newly stored in the table storage unit 45.
Above, the label production | generation process which the image process part 4 performs is complete | finished.

As described above, the imaging apparatus 100 according to the present embodiment can extract the main color, which is the feature amount of the image data, with a small amount of calculation compared to the related art. Furthermore, the imaging apparatus 100 according to the present embodiment has a small number of scenes because the scene is determined using information included in Exif and the table for each scene stored in the table storage unit 45 is selected based on the determination result. Scenes can be identified by the amount of computation. As a result, the imaging apparatus 100 according to the present embodiment can generate a large number of labels with less arithmetic processing and fewer options for image data than in the related art.
That is, the image processing unit 4 extracts three main colors having a high frequency from the color vector obtained by converting the image data into the color space, and stores the first label stored in advance in association with the extracted main colors. Extract. As shown in FIGS. 10 and 11, since the first label is stored in advance in association with the main color for each scene, each time, and each season, the image processing unit 4 is extracted from the image data. Even if the main colors are the same, different first labels can be generated for each scene, time, and season, so that an optimum label for image data can be generated for each scene.
Further, the image processing unit 4 normalizes the frequencies of the three main colors, generates a modified label that modifies the generated first label according to the ratio of the most frequent first color, and generates the generated modification The first label is modified by modifying the first label with the label to generate a second label.
As a result, the image processing unit 4 generates the second label by modifying the first label with the modification label based on the ratio of the color arrangement of the main color of the image data. Compared with the case where labels are generated by extracting colors, a more optimal label can be generated for image data for each scene.

In the present embodiment, the color space vector generation unit 43 has described an example in which the image data is generated in the HSV color space. However, RGB (red, green, blue), a luminance signal, and two color difference signals are described. YCrCb or YPbPr, HLS based on hue and saturation and lightness, Lab which is a kind of complementary color space, a color space based on the Japan Color Coordinating System (PCCS), and the like may be used.
In the present embodiment, the color space vector generation unit 43 generates a color vector frequency distribution and outputs information indicating the generated color vector frequency distribution to the main color extraction unit 44. The color space vector generation unit 43 may detect the frequency for each color vector and output information indicating the frequency for each detected color vector to the main color extraction unit 44. Also in this case, for example, each RGB value stored in the table storage unit 45 may be a color selected by the table creator from an interval of every 1 or 10 or the like.

In the present embodiment, the label output unit 48 has described the example in which the information indicating the label is stored in the table storage unit 45 in association with the image data. However, the label output by the second label generation unit 47 is the character information ( As text data, the data may be displayed on the display unit 7 so as to overlap the image data selected by the user.
Moreover, although the 1st label and the 2nd label demonstrated the example of an adjective or an adverb in this embodiment, the 1st label and the 2nd label may be a noun, for example. In this case, the first label is, for example, “exhilarating”, “rejuvenation”, “dandy”, and the like.

  In the present embodiment, an example in which the main color is calculated from the image data has been described. However, the main color extraction unit 44 extracts three colors in which adjacent color vectors are separated by a predetermined distance. It may be. In FIG. 7B, the adjacent color vectors are, for example, color vectors (50, 50, 50) and (50, 50, 51) when the image data is HSV. The distance between adjacent colors may be set based on a known threshold value that can identify a human visual color. For example, WEB256 colors recommended for use in WEB and monotone 256 colors that can be expressed in black and white may be used.

In addition, the main color extraction unit 44 performs a smoothing process on the frequency distribution of the color vectors generated by the color space vector generation unit 43 using a known method before calculating the main colors. Also good. Alternatively, the color space vector generation unit 43 may perform a color reduction process using a known method before generating the color space vector. For example, the color space vector generation unit 43 may reduce the image data to WEB color.
Further, in the present embodiment, an example has been described in which the main color extraction unit 44 extracts three frequently used colors from image data as main colors. However, the number of colors to be extracted is not limited to three, and two or more colors. If it is.

  In the present embodiment, an example in which HSV is used as a color vector has been described. When storing the combination of three colors as shown in FIG. 4 in the table storage unit 45, HSV = (0,0,0), (1,0, 0), (1, 1, 0)... (100, 100, 99), (100, 100, 100) may be selected by the table creator. Alternatively, HSV = (0,0,0), (10,0,0), (10,10,0) (100, 100, 90), (100, 90, etc.) 100, 100, 100) may be selected by the table creator. In this way, by setting the interval of each value in the color vector to a predetermined value such as 10 or the like, the capacity to be stored in the table storage unit 45 can be reduced, and the calculation amount can also be reduced.

[Second Embodiment]
In the first embodiment, the example in which the scene of the image data selected by the user is determined based on the image identification information stored in the storage medium 200 in association with the image data has been described. In the present embodiment, an example will be described in which an image processing apparatus discriminates a scene from selected image data and generates a label based on the discrimination result.

FIG. 13 is a block diagram of the image processing unit 4a according to the present embodiment.
As shown in FIG. 13, the image processing unit 4a includes an image acquisition unit 41a, an image identification information acquisition unit 42, a color space vector generation unit 43, a main color extraction unit 44, a table storage unit 45, a first label generation unit 46a, A second label generation unit 47, a label output unit 48, a feature amount extraction unit 241, and a scene determination unit 242 are provided. Note that functional units having the same functions as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  The image acquisition unit 41 a reads the image data captured by the imaging unit 2 and the image identification information stored in association with the image data from the storage medium 200 via the bus 15. The image acquisition unit 41a outputs the acquired image data to the color space vector generation unit 43 and the feature amount extraction unit 241. The image acquisition unit 41 a outputs the acquired image identification information to the image identification information acquisition unit 42.

  The feature amount extraction unit 241 extracts feature amounts from the image data output from the image acquisition unit 41a by a known method. Known methods use, for example, methods such as image binarization, smoothing, edge detection, and contour detection. The feature amount extraction unit 241 outputs information indicating the extracted feature amount to the scene determination unit 242.

The scene discrimination unit 242 discriminates the scene of the image data acquired by the image acquisition unit 41a using a known method based on the information indicating the feature amount output from the feature amount extraction unit 241. The known method used for scene discrimination is, for example, as in the prior art described in Patent Document 1, the scene discrimination unit 242 divides image data into a plurality of predetermined areas, and each area is divided. Based on the feature amount, it is determined whether a person is reflected in the image data or whether the sky is reflected. Then, based on the determination result, the scene determination unit 242 determines the scene of the image data.
The scene determination unit 242 outputs information indicating the determined scene to the first label generation unit 46a.
In the present embodiment, the scene determination unit 242 may be configured by a feature amount extraction unit 241 and a scene determination unit 242.

  The first label generation unit 46a stores the first label stored in association with the information indicating the scene output from the scene determination unit 242 and the information indicating the main color output from the main color extraction unit 44. Read from 45. The first label generation unit 46 a outputs the information indicating the read first label and the information indicating the main color output by the main color extraction unit 44 to the second label generation unit 47.

  Next, label generation processing performed by the image processing unit 4a of the imaging apparatus 100 will be described with reference to FIG. The imaging apparatus 100 performs step S1 and step S2 as in the first embodiment.

(Step S3) Next, the feature quantity extraction unit 241 extracts a feature quantity from the image data output by the image acquisition unit 41a by a known method, and outputs information indicating the extracted feature quantity to the scene determination unit 242. .
Next, the scene discriminating unit 242 extracts a scene, which is shooting information of the image data acquired by the image acquiring unit 41a, using a known method based on the information indicating the feature amount output by the feature amount extracting unit 241. The information indicating the acquired scene is output to the first label generation unit 46a. After step S3 ends, the process proceeds to step S4.

  The image processing unit 4a performs step S4 and step S5 as in the first embodiment. After step S5 ends, the process proceeds to step S6.

  (Step S6) Next, the first label generation unit 46a stores the information indicating the scene output from the scene determination unit 242 and the information indicating the main color output from the main color extraction unit 44 in association with each other. One label is read from the table storage unit 45. Next, the first label generation unit 46 a outputs the information indicating the read first label and the information indicating the main color output by the main color extraction unit 44 to the second label generation unit 47. After step S6 ends, the image processing unit 4a performs steps S7 to S9 as in the first embodiment.

  As described above, the image processing unit 4a determines the scene of the captured image data by a predetermined method, and performs the first implementation based on the determined scene and the three main colors extracted from the image data. The label is generated in the same way as the form. As a result, the image processing unit 4a can generate an optimum label for the image data even when the image identification information is not stored in the storage medium 200 in association with the image data.

In the present embodiment, the example in which the image processing unit 4a generates the label based on the scene determined from the image data and the extracted main color has been described. However, as in the first embodiment, the shooting information is also generated. It may be used to determine the scene. For example, the image processing unit 4a may extract information indicating the date and time of image capture from the image identification information, and generate a label based on the extracted image capture date and scene determined from the image data. More specifically, when the scene is “landscape” and the imaging date is “autumn”, the first label stored in association with the scenes “landscape”, “autumn”, and main colors is read out and read out. A label may be generated based on the two first labels.
Alternatively, the main color and the first label may be stored in the table storage unit 45 with the scene as “autumn scenery”.

[Third Embodiment]
In the first embodiment and the second embodiment, the example in which the label is generated based on the main color extracted from the entire image data selected by the user has been described. In the present embodiment, an example will be described in which a scene is determined from selected image data, a main color is extracted in a predetermined image data area based on the determined scene, and a label is generated from the extracted main color. .

FIG. 14 is a block diagram of the image processing unit 4b according to the embodiment of the present embodiment.
As shown in FIG. 14, the image processing unit 4b includes an image acquisition unit 41b, an image identification information acquisition unit 42b, a color space vector generation unit 43b, a main color extraction unit 44, a table storage unit 45, a first label generation unit 46, A second label generation unit 47, a label output unit 48, and a region extraction unit 341 are provided. Note that functional units having the same functions as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  The image acquisition unit 41 b reads the image data captured by the imaging unit 2 and the image identification information stored in association with the image data from the storage medium 200 via the bus 15. The image acquisition unit 41b outputs the acquired image data to the region extraction unit 341 and the color space vector generation unit 43b. The image acquisition unit 41b outputs the acquired image identification information to the image identification information acquisition unit 42b.

  The image identification information acquisition unit 42b extracts shooting information set in the captured image data from the image identification information output by the image acquisition unit 41b, and uses the extracted shooting information as a first label generation unit 46 and a region extraction unit. 341.

The region extraction unit 341 extracts a region for extracting a main color from the image data output from the image identification information acquisition unit 42b based on the shooting information output from the image identification information acquisition unit 42b. The region extraction unit 341 extracts image data of a region for extracting the extracted main color from the image data output by the image identification information acquisition unit 42b, and outputs the image data of the extracted region to the color space vector generation unit 43b. .
Note that, as a technique for extracting a predetermined region for extracting a main color, for example, a region to be extracted from the entire image may be set in advance for each scene. For example, when the scene is “landscape”, the area is two-thirds from the top of the image data, and when the scene is “portrait”, the area has a predetermined size at the center of the image data.
Alternatively, in combination with the second embodiment, based on the feature amount extracted from the image data, the region from which the feature amount is extracted may be extracted as a region for extracting the main color. In this case, a plurality of areas may be extracted from the image data. For example, when it is determined that the scene of the captured image data is a portrait, the scene determination unit 242 in FIG. 13 performs face detection using a technique such as feature amount extraction. When there are a plurality of detected face regions, the scene determination unit 242 detects the main color from each of the detected regions. Then, the first label generation unit 46 and the second label generation unit 47 may generate a plurality of labels for each detected main color. Alternatively, the scene determination unit 242 may output the determination result to the main color extraction unit 44 so as to use an area including all detected face areas as an area for extracting the main color.

Returning to FIG. 14, the color space vector generation unit 43b converts the image data output from the region extraction unit 341 into a vector of a predetermined color space. The predetermined color space is, for example, HSV. The color space vector generation unit 43b classifies all pixels of the image data for each generated color vector, detects the frequency for each color vector, and generates a color vector frequency distribution.
The color space vector generation unit 43 b outputs information indicating the frequency distribution of the generated color vector to the main color extraction unit 44.

  Next, label generation processing performed by the image processing unit 4b of the imaging apparatus 100 will be described with reference to FIG. FIG. 15 is a flowchart of label generation performed by the imaging apparatus 100 according to the present embodiment. The imaging apparatus 100 performs step S1 as in the first embodiment. After step S1, the process proceeds to step S101.

(Step S101) Next, the image acquisition unit 41b of the image processing unit 4b stores the image data captured by the imaging unit 2 via the bus 15 and the image identification information stored in association with the image data. Read from the medium 200.
Next, the image acquisition unit 41b outputs the acquired image data to the region extraction unit 341 and the color space vector generation unit 43b. Next, the image acquisition unit 41b outputs the acquired image identification information to the image identification information acquisition unit 42b. After step S101 ends, the process proceeds to step S3.

  (Step S3) The image processing unit 4b performs step S3 as in the first embodiment. After step S3 ends, the process proceeds to step S102.

(Step S102) Next, the region extraction unit 341 selects a main color from the image data output from the image identification information acquisition unit 42b based on the shooting information output from the image identification information acquisition unit 42b by a predetermined method. Extract the area to be extracted.
Next, the area extraction unit 341 extracts image data of an area from which the extracted main color is extracted from the image data output by the image identification information acquisition unit 42b, and the image data of the extracted area is used as a color space vector generation unit 43b. Output to. After step S102, the process proceeds to step S103.

  (Step S103) Next, the color space vector generation unit 43b converts the image data of the region output from the region extraction unit 341 into a vector of a predetermined color space. Next, the color space vector generation unit 43b classifies all pixels of the image data for each generated color vector, detects the frequency for each color vector, and generates a color vector frequency distribution. Next, the color space vector generation unit 43 b outputs information indicating the frequency distribution of the generated color vector to the main color extraction unit 44. After step S103 ends, the process proceeds to step S5.

  Hereinafter, the image processing unit 4b performs steps S5 to S9 in the same manner as in the first embodiment.

  As described above, the image processing unit 4b extracts a region for extracting a main color from captured image data based on shooting information such as a scene. Then, the image processing unit 4b generates a label in the same manner as in the first embodiment, based on the three main colors extracted from the image data of the region from which the main color is extracted. As a result, the image processing unit 4b extracts the main color from the image data of the area corresponding to the scene, and generates the label based on the extracted main color of the area. Therefore, the first embodiment and the second embodiment. As compared with the above, it is possible to generate an optimum label for image data suitable for a scene.

[Fourth Embodiment]
In the first to third embodiments, the example in which three colors are selected as the main colors from the image data selected by the user has been described. In the present embodiment, an example in which three or more colors are selected from selected image data will be described. The configuration of the image processing unit 4 will be described for the same case as in the first embodiment (FIG. 2).

FIG. 16 is a diagram illustrating an example of extracting a plurality of color vectors from image data according to the present embodiment. In FIG. 16, the horizontal axis represents the color vector, and the vertical axis represents the frequency.
In FIG. 16, it is assumed that the main color extraction unit 44 has extracted the first color vector c21, the second color vector c22, and the third color vector c23 as in FIG. 8B.
In FIG. 16, when the frequency of the color vectors c24, c25, c26 is within a predetermined range, the main color extraction unit 44 extracts the color vectors c24, c25, c26 as the fourth main color. In this case, the table storage unit 45 stores a label for each scene including the fourth color in addition to the first to third colors described in FIG.
When the fourth color is extracted, the main color extraction unit 44 reads out the first label of the combination of the first color to the fourth color stored in the table storage unit 45 and stores the first label stored therein. To extract. When a plurality of first labels of combinations of the first color to the fourth color are stored, the main color extraction unit 44 selects, for example, the first label read from the table storage unit 45 first. Or you may make it select at random.

In addition, the main color extraction unit 44 may select three colors as main colors from the extracted four colors. In this case, the main color extraction unit 44 may calculate the degree of approximation of the four extracted colors and calculate the three colors having low degrees of approximation as the main colors. For example, in FIG. 16, the color approximation degree will be described assuming that four color vectors c22 to c25 are extracted as the first to fourth colors. The main color extraction unit 44 reduces the four extracted colors from an 8-bit color space to, for example, a 7-bit color space. After color reduction, for example, when the color vectors c24 and c25 are determined to be the same color, the main color extraction unit 44 determines that the color vectors c24 and c25 are approximate colors. Then, the main color extraction unit 44 selects one of the color vectors c24 and c25 as the third main color. In this case, in the frequency distribution of FIG. 16, the main color extraction unit 44 selects the color vector having the larger distance away from the first color vector c22 and the second color vector c23 in the horizontal axis direction. You may make it like, You may select at random.
If the four color vectors remain separated even after the color is reduced to the 7-bit color space, the color space vector generation unit 43 performs the color reduction until the four color vectors are integrated into the three color vectors.

As described above, four or more main colors and first labels are stored in advance in the table storage unit 45 for each scene as shooting information, and four or more main colors are extracted from the image data. Since the label is generated based on the extracted main color and the scene, it is possible to generate an optimum label for the image data as compared with the first to third embodiments.
That is, in the present embodiment, the image processing unit 4 extracts four frequently used colors from the color vector obtained by converting the image data into the color space, and stores them in association with the four extracted colors in advance. Extract the first label. Since the first label is stored in association with the four extracted main color vectors in advance for each shooting information, for example, for each scene, for each time or season, the image processing unit 4 is extracted from the image data. Even if the main colors are the same, different first labels can be generated for each scene, time, and season. Further, the image processing unit 4 normalizes the frequencies of the four main colors, and adds a second label that emphasizes the first label to the generated first label according to the ratio of the most frequent first color. Appends to generate a label. As a result, the image processing unit 4 can generate an optimum label for the image data based on the four main colors as compared with the first to third embodiments.
In addition, the image processing unit 4 extracts three main colors from the four extracted main colors by subtracting colors, and performs label generation processing on the extracted three main colors in the same manner as in the first embodiment. Do. As a result, the image processing unit 4 can generate an optimum label for the image data even if the image data has a small difference in color vector frequency.

In this embodiment, an example in which four main colors are extracted from image data has been described. However, the main colors to be extracted are not limited to four colors, and may be more than that. In this case, the first label corresponding to the number of extracted main colors may be stored in the table storage unit 45. For example, when extracting five main colors, as described above, the main color extraction unit 44 performs subtractive color integration to approximate colors, and extracts three main colors from the extracted plurality of main colors. You may make it extract again. Further, for example, when extracting six main colors, the main color extracting unit 44 first sets the first group of the first to third colors and the remaining fourth to sixth colors in descending order of frequency. Separate into a second group. The fourth color has fewer pixels than the third color and more pixels than the fifth color, and the fifth color has fewer pixels than the fourth color.
Then, the first label generation unit 46 extracts the first label corresponding to the first group and the first label corresponding to the second group. And the 1st label production | generation part 46 carries out a 1st label with a decoration label about the two 1st labels extracted in this way according to the frequency of the 1st color or the 4th color like a 1st embodiment. It may be modified by modification to generate a plurality of labels. Alternatively, the second label generation unit 47 may integrate a plurality of labels generated in this way to generate one label. Specifically, when the label by the first group is “very refreshing” and the label by the second group is “a little childish”, the second label generation unit 47 “is very refreshing and a little childish”. May be generated. In the case of generating such two labels, the second label generating unit 47 determines which of the two labels is arranged first in the second label generating unit 47 and can generate an appropriate label. For confirmation, a processing function unit that performs a language analysis process (not shown) may be provided.

  In the first to fourth embodiments, an example in which one label is generated for one image data has been described. However, two or more labels may be generated. In this case, for example, the color space vector generation unit 43 (including 43b) generates a color vector frequency distribution for each divided region divided into an upper half and a lower half in the image data of FIG. The main color extraction unit 44 extracts each main color by three colors from the frequency vector frequency distribution for each divided area. Then, the first label generation unit 46 may extract a label for each region from the table storage unit 45. Then, the label output unit 48 may store the plurality of labels generated in this manner in the storage medium 200 in association with the image data.

In the first to third embodiments, the example in which the three main colors and the first label are associated with each scene and stored in the table storage unit 45 has been described. However, for example, a single color for each scene And the first label may be associated with each other and stored in the table storage unit 45. In this case, as described in the first embodiment, the table storage unit 45 stores the three main colors and the first label in association with each scene, and further stores the single color and the first label for each scene. You may make it memorize | store in association.
By such processing, an appropriate label can be generated even for image data in which the image data is monotone and only one main color can be extracted. In this case, for example, the image processing unit 4 (4a, 4b) detects four colors as main colors as in the fourth embodiment, and only the first group of the first to third colors and the remaining fourth colors. The label may be read from the table storage unit 45 as a single color.
Further, when the color tone of the image data is monotonous and only two main colors can be extracted, for example, the first label generation unit 46 has two extracted main colors (first color and second color). Read each first label. Next, the second label generation unit 47 normalizes the two main colors based on the frequency of the extracted two main colors, and generates a modified label for the first color label based on the ratio of the first colors. Then, the second label of the first color may be generated by modifying and correcting the first label of the first color with the generated modified label. Or the 2nd label production | generation part 47 produces | generates two labels, the 1st label of the 1st color produced | generated in this way, and the 1st label of the 2nd color, or the 1st label of the 1st color And the first label of the second color may be integrated to generate one label.

  In the first to fourth embodiments, the example in which the image data selected by the user is read from the storage medium 200 has been described. However, the image data used for the label generation process is stored in the storage medium 200 as raw (RAW) data. When data and JPEG (Joint Photographic Experts Group) data are recorded, either RAW data or JPEG data may be used. Further, when thumbnail image data reduced for display on the display unit 7 is stored in the storage medium 200, a label may be generated using the thumbnail image data. Even if the thumbnail image data is not stored in the storage medium, the color space vector generation unit 43 (including 43b) determines the resolution of the image data output from the image acquisition unit 41 (including 41a and 41b) in advance. Image data reduced in resolution may be generated, and the frequency of color vectors and main colors may be extracted from the reduced image data.

Note that a program for realizing the function of each unit of the image processing unit 4 in FIG. 2, the image processing unit 4a in FIG. 13, or the image processing unit 4b in FIG. 14 is recorded on a computer-readable recording medium. The program recorded on the recording medium may be read into a computer system and executed to execute the processing of each unit. Here, the “computer system” includes an OS and hardware such as peripheral devices.
Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” is a portable medium such as a flexible disk, a magneto-optical disk, a ROM (Read Only Memory), a CD-ROM, or a USB (Universal Serial Bus) I / F (interface). A storage device such as a USB memory or a hard disk built in a computer system. Further, the “computer-readable recording medium” includes a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system serving as a server or a client. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  FIG. 17 is a diagram schematically illustrating an example of a process for extracting a feature amount of a captured image used for determining a sentence to be arranged on an image. In the example of FIG. 17, the determination unit of the image processing apparatus classifies the scene of the captured image into a person image or a landscape image. Next, the image processing apparatus extracts a feature amount of the captured image according to the scene. The feature amount can be the number of faces (number of subjects) and the average color (color arrangement pattern) in the case of a person image, and can be the average color (color arrangement pattern) in the case of a landscape image. . Based on these feature quantities, words (adjectives and the like) to be inserted into the person image template or the landscape image template are determined.

  Here, in the example of FIG. 17, the color arrangement pattern is configured by a combination of a plurality of representative colors that form the captured image. Therefore, the color arrangement pattern can represent the average color (average color) of the captured image. In one example, “first color”, “second color”, and “third color” are defined as a color arrangement pattern, and based on a combination of these three colors, that is, based on three average colors, a person image Or a word (adjective) to be inserted into a text template for a landscape image.

  In the example of FIG. 17, the scene of the captured image is classified into two types (a person image and a landscape image). In another example, the scene of the captured image can be classified into three or more types (3, 4, 5, 6, 7, 8, 9, or 10 types or more).

  FIG. 18 is a diagram schematically illustrating another example of the process of extracting the feature amount of the captured image used for determining the text arranged on the image. In the example of FIG. 18, the scene of the captured image can be classified into three or more types.

  In the example of FIG. 18, the determination unit of the image processing apparatus determines whether the captured image is a person image (first mode image), a distant view image (second mode image), or another image (third mode image). judge. First, as in the example of FIG. 17, the determination unit determines whether the captured image is a person image or an image different from the person image.

  Next, when the captured image is an image different from the human image, the determination unit determines whether the captured image is a distant view image (second mode image) or another image (third mode image). To do. This determination can be performed using, for example, a part of the image identification information given to the captured image.

  Specifically, in order to determine whether the captured image is a distant view image, a focal length that is a part of the image identification information can be used. The determination unit determines that the captured image is a distant view image when the focal distance is greater than or equal to a preset reference distance, and determines the captured image as another image when the focal distance is less than the reference distance. As described above, the captured image is classified into three types of scenes: a person image (first mode image), a distant view image (second mode image), or another image (third mode image). Note that examples of distant view images (second mode images) include landscape images such as the sea and mountains, and examples of other images (third mode images) include flowers and pets.

  Also in the example of FIG. 18, after the scene of the captured image is classified, the image processing apparatus extracts the feature amount of the captured image according to the scene.

  In the example of FIG. 18, when the captured image is a person image (first scene image), the number of faces (number of subjects) and the feature amount of the captured image used to determine the text arranged on the image A smile level can be used. That is, when the captured image is a human image, a word to be inserted into the human image template may be determined based on the determination result of the smile level in addition to or instead of the determination result of the number of faces (number of subjects). it can. Hereinafter, an example of a smile level determination method will be described with reference to FIG.

  In the example of FIG. 19, the determination unit of the image processing apparatus detects a face area from a human image by a method such as face recognition (step S5001). In one example, the degree of smile of a person image is calculated by digitizing the degree of ascending of the mouth corner. For example, various known techniques for face recognition can be used for calculating the smile level.

  Next, the determination unit compares the smile level with the first smile threshold α set in advance (step S5002). When it is determined that the smile level is greater than or equal to α, the determination unit determines that the smile level of the person image is “smile: large”.

  On the other hand, when it is determined that the smile level is less than α, the determination unit compares the smile level with a second smile threshold value β set in advance (step S5003). When it is determined that the smile level is β or more, the determination unit determines that the smile level of this person image is “smile: medium”. Furthermore, when it is determined that the smile level is less than β, the determination unit determines that the smile level of the person image is “smile: small”.

  A word to be inserted into the person image template is determined based on the determination result of the smile level of the person image. Here, examples of the word corresponding to the smile level of “smile: large” include “full of joy” and “very good”. Examples of words that correspond to the smile level of “smile: medium” include “joyful” and “good calm”. Examples of words corresponding to the smile level of “smile: small” include “seriously seems” and “cool”.

  In the above description, the case where the word inserted into the person image template is a continuous form has been described. However, the present invention is not limited to this, and may be an end form, for example. In this case, examples of words corresponding to the smile level of “smile: large” include “smile is nice”, “it is a very good smile”, and the like. Examples of words corresponding to the smile level of “smile: medium” include “smiley” and “good expression”. Examples of words corresponding to a smile level of “smile: small” include “looks serious” and “looks serious”.

  FIG. 20A is an example of an output image showing the operation result of the image processing apparatus, and this output image has a sentence determined based on the example of FIG. In the example of FIG. 20A, the captured image is determined to be a person image, and the number of subjects and the color arrangement pattern (average color) are extracted as the feature amount. Further, the word inserted into the person image template is determined as “heavy” according to the color arrangement pattern. As a result, the output result shown in FIG. 20A is obtained. That is, in the example of FIG. 20A, the word “heavy” (adjective, combined form) is determined based on the average color of the captured image.

  FIG. 20B is another example of an output image showing the operation result of the image processing apparatus, and this output image has a sentence determined based on the example of FIG. In the example of FIG. 20B, the captured image is determined to be a person image, and the number of subjects and the smile level are extracted as the feature amount. Further, according to the smile level, the word inserted into the person image template is determined as “good expression”. As a result, the output result shown in FIG. 20B is obtained. That is, in the example of FIG. 20B, the word (end form) of “good expression” is determined based on the smile level of the person in the captured image. As shown in the output result of FIG. 20B, by using a word output using a smile level for a person image, it is possible to attach character information that is relatively close to the impression received from the image.

  Returning to FIG. 18, when the captured image is a landscape image (second scene image) or another image (third scene image), as a feature amount of the captured image used to determine the text arranged on the image, A representative color can be used instead of the average color. As the representative color, the “first color” in the color arrangement pattern, that is, the most frequently used color in the captured image can be used. Alternatively, the representative color can be determined using clustering as described below.

  FIG. 21 is a schematic block diagram illustrating an internal configuration of an image processing unit included in the imaging apparatus. In the example of FIG. 21, the image processing unit 5040 of the image processing apparatus includes an image data input unit 5042, an analysis unit 5044, a text creation unit 5052, and a text addition unit 5054. The image processing unit 5040 performs various types of analysis processing on the image data generated by the imaging unit or the like, thereby acquiring various types of information regarding the content of the image data, and creating text that is highly consistent with the content of the image data. Then, text can be added to the image data.

  The analysis unit 5044 includes a color information extraction unit 5046, a region extraction unit 5048, and a clustering unit 5050, and performs analysis processing on the image data. The color information extraction unit 5046 extracts first information regarding color information of each pixel included in the image data from the image data. Typically, the first information is a total of the HSV values of all the pixels included in the image data. However, the first information is the frequency at which the predetermined color appears in the image (frequency in pixel units, area ratio, etc.) for a predetermined color associated with similarity (for example, associated with a predetermined color space). The color resolution and the type of color space are not limited.

  For example, the first information may be information indicating how many pixels of each color are included in the image data for each color represented by an HSV space vector (HSV value) or RGB value. . However, the color resolution in the first information may be changed as appropriate in consideration of the burden of calculation processing, and the type of color space is not limited to HSV or RGB, and may be CMY, CMYK, or the like.

  FIG. 22 is a flowchart showing the flow of representative color determination performed in the analysis unit 5044. In step S5101, the image processing apparatus starts calculating the representative color of specific image data 5060 (captured image, see FIG. 23).

  In step S5102, the image data input unit 5042 of the image processing apparatus outputs the image data to the analysis unit 5044. Next, the color information extraction unit 5046 of the analysis unit 5044 calculates first information 5062 regarding the color information of each pixel included in the image data (see FIG. 23).

  FIG. 23 is a conceptual diagram illustrating a calculation process of the first information 5062 performed by the color information extraction unit 5046 in step S5102. The color information extraction unit 5046 aggregates the color information included in the image data 5060 for each color (for example, for each gradation of 256 gradations) to obtain first information 5062. The histogram shown in the lower part of FIG. 23 represents an image of the first information 5062 calculated by the color information extraction unit 5046. The horizontal axis of the histogram in FIG. 23 is color, and the vertical axis represents how many pixels of a predetermined color are included in the image data 5060.

  In step S5103 of FIG. 22, the region extraction unit 5048 of the analysis unit 5044 extracts the main region in the image data 5060. For example, the area extraction unit 5048 extracts a focused area from the image data 5060 shown in FIG. 23, and recognizes the central portion of the image data 5060 as the main area (see the main area 5064 in FIG. 24). ).

  In step S5104 in FIG. 22, the region extraction unit 5048 of the analysis unit 5044 determines a clustering target region to be implemented in step S5105. For example, as shown in the upper part of FIG. 24, the area extraction unit 5048 recognizes that part of the image data 5060 is the main area 5064 in step S5103 and extracts the main area 5064, the clustering target is set as the main area 5064. The first information 5062 (main first information 5066) corresponding to the area 5064 is used. The histogram shown in the lower part of FIG. 24 represents an image of the main first information 5066.

  On the other hand, if the region extraction unit 5048 has not extracted the main region 5064 in the image data 5060 in step S5103, the region extraction unit 5048 displays the first information corresponding to the entire region of the image data 5060 as shown in FIG. 5062 is determined as a clustering target. Note that there is no difference in the subsequent processing between the case where the main region 5064 is extracted and the case where it is not extracted, except that the target region for clustering is different. I will explain.

  In step S5105 of FIG. 22, the clustering unit 5050 of the analysis unit 5044 performs clustering on the main first information 5066 that is the first information 5062 of the area determined in step S5104. FIG. 25 is a conceptual diagram illustrating the result of clustering performed by the clustering unit 5050 on the primary first information 5066 of the primary region 5064 shown in FIG.

  For example, the clustering unit 5050 classifies the main first information 5066 (see FIG. 24) having 256 gradations into a plurality of clusters by the k-means method. Note that the clustering is not limited to the k-means method (k average method). In other examples, other methods such as the shortest distance method can be used.

  The upper part of FIG. 25 shows to which cluster each pixel is classified, and the histogram shown in the lower part of FIG. 25 shows the number of pixels belonging to each cluster. By the clustering by the clustering unit 5050, the 256 first main information 5066 (FIG. 24) is classified into less than 256 clusters (three in the example shown in FIG. 25). The result of clustering can include information about the size of each cluster and information about the color of each cluster (the position of the cluster in the color space).

  In step S5106, the clustering unit 5050 of the analysis unit 5044 determines a representative color of the image data 5060 based on the clustering result. In one example, when the clustering unit 5050 obtains a clustering result as illustrated in FIG. 25, the color belonging to the maximum cluster 5074 including the most pixels among the plurality of calculated clusters is set as the representative color of the image data 5060. To do.

  When the calculation of the representative color is completed, the sentence creation unit 5052 creates a text using information on the representative color and assigns the text to the image data 5060.

  The text creation unit 5052 reads a text template for a landscape image, for example, and applies a word (for example, “2012/03/10”) corresponding to the generation date and time of the image data 5060 to {date and time} of the text template. In this case, the analysis unit 5044 can retrieve information related to the generation date and time of the image data 5060 from the storage medium and output the information to the text creation unit 5052.

  In addition, the sentence creation unit 5052 applies a word corresponding to the representative color of the image data 5060 to the {adjective} of the sentence template. The sentence creation unit 5052 reads the correspondence information from the storage unit 5028 and applies it to the sentence template. In one example, the storage unit 5028 stores a table in which colors and words are associated with each scene. The sentence creation unit 5052 can create a sentence (for example, “I found a very beautiful thing”) using words read from the table.

  FIG. 26 shows image data 5080 to which text is given by the series of processes described above.

  FIG. 27 shows an example of image data to which text is given by a series of processes similar to the above when the scene is a distant view image. In this case, the scene is classified as a distant view image, and the representative color is determined to be blue. For example, in a table in which colors and words are associated with each scene, the word “fresh” is associated with the representative color “blue”.

  FIG. 28 is a diagram illustrating an example of a table having correspondence information between colors and words. In the table of FIG. 28, a color and a word are associated with each scene of a person image (first scene image), a distant view image (second scene image), and another image (third scene image). In one example, when the representative color of the image data is “blue” and the scene is another image (third scene image), the sentence creation unit 5052 uses a word corresponding to the representative color (for example, from the correspondence information in the table). “Classy”) and select {adjective} in the sentence template.

  The correspondence table between colors and words can be set based on a color chart such as a PCCS color system, CICC color system, or NCS color system.

  FIG. 29 shows an example of a correspondence table for a distant view image (second scene image) using a color chart of the CCIC display system. FIG. 30 shows an example of a correspondence table for other images (third scene images) using a CCIC display color chart.

  In FIG. 29, the horizontal axis corresponds to the hue of the representative color, and the vertical axis corresponds to the tone of the representative color. By using the table of FIG. 29 to determine the word, not only the information on the hue of the representative color but also the information on the tone of the representative color is used to determine the word, and give a text that is relatively close to the sensibility generated by humans Is possible. Hereinafter, a specific text setting example in the case of a distant view image (second scene image) using the table of FIG. 29 will be described. In the case of other images (third scene images), the same setting can be made using the table of FIG.

  In FIG. 29, when the representative color is determined to be the area A5001, the name of the representative color (red, orange, yellow, blue, etc.) is applied as it is to the word in the text. For example, if the hue of the representative color is “red (R)” and the tone is “Vivid Tone (V)”, the adjective “crimson” representing the color is selected.

  When the representative color is determined to be the color of the region A5002, A5003, A5004, or A5005, an adjective associated with the color is applied to the word in the text. For example, when the representative color is determined to be the color (green) of the area A5003, the adjectives associated with green, such as “comfortable” and “fresh”, are applied.

  When the representative color is determined to be the color of the area A5001 to A5005 and the tone is a vivid tone (V), a strong tone (S), a bright tone (B), or a pale tone (LT). Applies adverbs that indicate the degree before the adjectives (eg, very, pretty, etc.).

  When the representative color is determined to be the area A5006, that is, “white tone (white)”, “clean”, “clear”, and the like, which are words associated with white, are selected. If the representative color is determined to be the area A5007, that is, a gray-based color (light gray tone: ltGY, medium gray tone: mGY, or dark gray tone: dkGY), it is a safe adjective. “Clean”, “nice”, etc. are selected. In an image in which a white or gray color, that is, an achromatic color is a representative color, various colors are often included in the entire image. Therefore, by using a word that is less related to color, it is possible to prevent a text having an inappropriate meaning from being added, and it is possible to provide a text that is relatively close to an image received from an image.

  Further, when the representative color does not belong to any of the areas A5001 to A5007, that is, when the representative color is a low tone (dark grayish tone) or black (black tone), characters having a predetermined meaning (Word or sentence) can be selected as text. Characters having a predetermined meaning include, for example, “where is here”, “a”, and the like. These words and sentences can be stored in the storage unit of the image processing apparatus as a “tweet dictionary”.

  In other words, when the representative color is determined to be low tone or black, it may be difficult to determine the hue of the entire image. Even in such a case, characters having less relation to the color are used as described above. Thus, it is possible to prevent a text having an inappropriate meaning from being added, and to add a text close to an image received from an image.

  In the above example, the case where the sentence and the word are uniquely determined according to the scene and the representative color has been described. However, the present invention is not limited to this, and exception processing is sometimes performed in the selection of the sentence and the word. it can. For example, the text may be extracted from the “tweet dictionary” once every plural times (for example, once every 10 times). As a result, the display content of the text is not necessarily patterned, so that the user can be prevented from getting bored with the display content.

  In the above example, the case where the sentence adding unit arranges the text generated by the sentence creating unit at the upper part or the lower part of the image has been described. However, the present invention is not limited to this. It can also be arranged.

  In the above example, the case where the position of the text is fixed in the image has been described. However, the present invention is not limited to this. For example, the text can be displayed so as to flow on the display unit of the image processing apparatus. Thereby, the input image is not easily affected by the text, or the text visibility is improved.

  In the above example, the case where the text is always pasted on the image has been described. However, the present invention is not limited to this. For example, in the case of a person image, the text is not pasted, and in the case of a distant view image or other images. You may make it paste a text.

  In the above example, the sentence adding unit has described the case where the display method (font, color, display position, etc.) of the text generated by the sentence creating unit is determined by a predetermined method. A variety of text display methods can be determined. Hereinafter, some examples of these methods will be described.

  In one example, the user can correct the text display method (font, color, display position) via the operation unit of the image processing apparatus. Alternatively, the user can change or delete the contents (words) of the text. In addition, the user can select not to display the entire text, that is, display / non-display of the text.

  In one example, the size of the text can be changed according to the scene of the input image. For example, when the scene of the input image is a person image, the text can be reduced, and when the scene of the input image is a distant view image or other images, the text can be increased.

  In one example, text can be highlighted and combined with image data. For example, when the input image is a person image, a balloon can be given to the person and text can be placed in the balloon.

  In one example, the display color of the text can be set with reference to the representative color of the input image. Specifically, a color having the same hue as the representative color of the input image and a different tone can be used as a text display color. As a result, it is possible to give a text that is in harmony with the input image without excessively claiming the text.

  In particular, when the representative color of the input image is white, exception processing may be performed in determining the text display color. Here, in the exception processing, for example, the text color can be set to white and the peripheral portion of the text can be set to black.

DESCRIPTION OF SYMBOLS 100 ... Imaging device, 1 ... Imaging system, 2 ... Imaging part, 3 ... Camera control part, 4, 4a, 4b ... Image processing part, 5 ... Memory | storage part, 6. ... Buffer memory section, 7... Display section, 11... Operation section, 12 .. Communication section, 13... Power supply section, 15. Image sensor, 23 ... AD conversion unit, 41, 41b ... image acquisition unit, 42, 42b ... image identification information acquisition unit, 43, 43b ... color space vector generation unit, 44 ... Main color extraction unit, 45 ... Table storage unit, 46, 46a ... First label generation unit, 47 ... Second label generation unit, 48 ... Label output unit, 241 ... Feature amount extraction , 242 ... Scene discriminator

Claims (10)

An image acquisition unit for acquiring captured image data;
A scene discriminating unit for discriminating a scene from the acquired image data;
A main color extraction unit for extracting a main color from the acquired image data based on a frequency distribution of color information;
A storage unit in which color information and a first label are associated and stored in advance for each scene;
Reading the first label stored in advance in association with the extracted main color and the determined scene from the storage unit, and generating the read first label as a label of the acquired image data A first label generator that
An image processing apparatus comprising: A second label generating unit that normalizes a ratio of the main color based on the frequency of the extracted main color and generates a second label by correcting the first label based on the normalized ratio of the main color. The image processing apparatus according to claim 1, further comprising: In the storage unit,
The image processing apparatus according to claim 1, wherein a combination information of a plurality of pieces of color information and a label are associated with each of the determined scenes. The scene discrimination unit
Acquiring image identification information from the acquired image data, extracting information indicating the scene from the acquired image identification information, and determining the scene of the image data based on the information indicating the extracted scene. The image processing apparatus according to any one of claims 1 to 3, wherein the image processing apparatus is characterized. The scene discrimination unit
The image processing apparatus according to claim 4, wherein a feature amount is extracted from the acquired image data, and the scene of the image data is determined based on the extracted feature amount. An area extracting unit for extracting an area for extracting the main color from the acquired image data based on the determined scene;
The main color extraction unit includes:
The image processing apparatus according to any one of claims 1 to 4, wherein the main color is extracted from image data of an area from which the main color is extracted. The first label and the second label generated by correcting the first label, or information based on the first label or the second label is associated with the acquired image data in the storage unit. The image processing apparatus according to any one of claims 2 to 6, wherein the image processing apparatus is stored.   An image pickup apparatus comprising the image processing apparatus according to claim 1. A program for causing a computer to execute image processing of an image processing apparatus having an imaging unit,
An image acquisition procedure for acquiring captured image data;
A scene determination procedure for determining a scene from the acquired image data;
A main color extraction procedure for extracting a main color from the acquired image data based on a frequency distribution of color information;
The extracted primary color, color information and first label for each scene are associated with each other, the first label is read from a storage unit that is stored in advance, and the read first label is used as the acquired image data. A first label generation procedure for generating as a label of
A program that causes a computer to execute. A scene discriminating unit for discriminating whether or not it is a person-photographed scene;
A color extraction unit that extracts color information from the image data when the scene determination unit determines that the scene is not a person-captured scene;
A storage unit in which color information and characters having a predetermined meaning are associated and stored in advance;
A reading unit that reads out, from the storage unit, characters having the predetermined meaning corresponding to the color information extracted by the color extraction unit when the scene determination unit determines that the scene is not a person-captured scene. A featured image processing apparatus.

Images