JP2008271058A - Information processing method, information processing apparatus, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a user to easily perform confirming operation. <P>SOLUTION: An information processing method according to the present invention is characterized in that scene information of image data is acquired from additional data added to the image data to discriminate a scene of an image that the image data represents based upon the image data, a scene that the scene information represents is compared with the discriminated scene, and when the scene information represents and the discriminated scene do not match each other, an image corrected based upon the scene that the scene information represents and an image corrected based upon the discriminated scene are shown to the user. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an information processing method, an information processing apparatus, and a program.

  Some digital still cameras have a mode setting dial for setting a shooting mode. When the user sets the shooting mode with the dial, the digital still camera determines shooting conditions (such as exposure time) according to the shooting mode and performs shooting. When shooting is performed, the digital still camera generates an image file. In this image file, additional data such as shooting conditions at the time of shooting is added to the image data of the shot image.

On the other hand, image processing is also performed on image data according to additional data. For example, when a printer performs printing based on an image file, the image data is corrected according to the shooting conditions indicated by the additional data, and printing is performed according to the corrected image data.
JP 2001-238177 A

When the digital still camera generates an image file, scene information corresponding to dial settings may be stored in the additional data. On the other hand, if the user forgets to set the shooting mode, scene information that does not match the content of the image data is stored in the additional data. For this reason, the scene of the image data may be identified by analyzing the image data without using the scene information of the additional data. However, when the scene indicated by the additional data does not match the scene of the identification result, it is desirable to confirm to the user.
It is an object of the present invention to simplify the user's confirmation work at the time of confirmation to the user.

The main invention for achieving the above object is to acquire scene information of image data from additional data added to the image data, identify a scene of an image indicated by the image data based on the image data, The scene indicated by the scene information is compared with the identified scene, and when the scene indicated by the scene information and the identified scene do not match, the image corrected based on the scene indicated by the scene information and the identified scene The corrected image is shown to the user.
Other features of the present invention will become apparent from the description of the present specification and the accompanying drawings.

At least the following matters will become clear from the description of the present specification and the accompanying drawings.
From the additional data added to the image data, obtain the scene information of the image data,
Based on the image data, the scene of the image indicated by the image data is identified,
Comparing the scene indicated by the scene information with the identified scene;
When the scene indicated by the scene information and the identified scene do not match, information indicating the image corrected based on the scene indicated by the scene information and the image corrected based on the identified scene is displayed to the user The processing method becomes clear.
According to such an information processing method, the user's confirmation work is simplified.

  Further, it is desirable to display a confirmation screen on the display unit when showing the corrected image to the user. Thereby, the user can confirm the effect of the correction process on the display unit.

  Moreover, it is desirable that the image corrected based on the scene indicated by the scene information and the image corrected based on the identified scene are displayed side by side on the confirmation screen. This makes it easier for the user to compare the effects of the correction process.

  In addition, the image indicated by the image data is displayed on the confirmation screen, a boundary is set at a predetermined position of the image indicated by the image data, and one side viewed from the boundary is set as a scene indicated by the scene information. Preferably, the image is corrected based on the image, and the other side as viewed from the boundary is an image corrected based on the identified scene. This makes it easier to compare two images at the boundary.

Further, when the identified scene is a scene indicating that there is a human face in the image,
The boundary is preferably set so as to pass through the face portion. Thereby, it can be easily determined whether or not the correction is appropriate.

  In addition, on the confirmation screen, one of the image corrected based on the scene indicated by the scene information and the image corrected based on the identified scene is displayed, and the other image is displayed according to a user instruction. It is desirable to switch and display. This makes it easier to compare the effects of correction processing over the entire range of the image.

  Further, when the corrected image is shown to the user, it is desirable to print on the medium an image corrected based on the scene indicated by the scene information and an image corrected based on the identified scene. This makes it easier for the user to obtain a print image that meets his / her wishes.

An information processing apparatus comprising a controller,
The controller is
From the additional data added to the image data, obtain the scene information of the image data,
Based on the image data, the scene of the image indicated by the image data is identified,
Comparing the scene indicated by the scene information with the identified scene;
When the scene indicated by the scene information and the identified scene do not match, information indicating the image corrected based on the scene indicated by the scene information and the image corrected based on the identified scene is displayed to the user The processing device becomes clear.
According to such an information processing apparatus, the user's confirmation work is simplified.

In the information processing device,
From the additional data added to the image data, the scene information of the image data is acquired,
Based on the image data, the scene of the image indicated by the image data is identified,
Comparing the scene indicated by the scene information with the identified scene;
When the scene indicated by the scene information and the identified scene do not match, the user is caused to indicate an image corrected based on the scene indicated by the scene information and an image corrected based on the identified scene. The program becomes clear.
According to such a program, the user's confirmation work is simplified.

=== Overall structure ===
FIG. 1 is an explanatory diagram of an image processing system. This image processing system includes a digital still camera 2 and a printer 4.

  The digital still camera 2 is a camera that acquires a digital image by forming an image of a subject on a digital device (CCD or the like). The digital still camera 2 is provided with a mode setting dial 2A. The user can set the shooting mode according to the shooting conditions by using the dial 2A. For example, when the “night scene” mode is set by the dial 2A, the digital still camera 2 performs shooting under shooting conditions suitable for night scene shooting by decreasing the shutter speed or increasing the ISO sensitivity.

  The digital still camera 2 stores an image file generated by photographing in the memory card 6 in accordance with the file format standard. In the image file, not only digital data (image data) of a captured image but also additional data such as a shooting condition (shooting data) at the time of shooting is stored.

  The printer 4 is a printing device that prints an image indicated by image data on paper. The printer 4 is provided with a slot 21 into which the memory card 6 is inserted. The user can take a picture with the digital still camera 2, remove the memory card 6 from the digital still camera 2, and insert the memory card 6 into the slot 21.

  The panel unit 15 includes a display unit 16 and an input unit 17 having various buttons. The display unit 16 is configured by a liquid crystal display. If the display unit 16 is a touch panel, the display unit 16 also functions as the input unit 17. The display unit 16 displays a setting screen for setting the printer 4, an image of image data read from the memory card, a screen for confirmation and warning to the user, and the like. Various screens displayed by the display unit 16 will be described later.

  FIG. 2 is an explanatory diagram of the configuration of the printer 4. The printer 4 includes a printing mechanism 10 and a printer-side controller 20 that controls the printing mechanism 10. The printing mechanism 10 includes a head 11 that ejects ink, a head control unit 12 that controls the head 11, a motor 13 for conveying paper, and a sensor 14. The printer-side controller 20 includes a memory slot 21 for transmitting and receiving data from the memory card 6, a CPU 22, a memory 23, a control unit 24 for controlling the motor 13, and a drive signal for generating a drive signal (drive waveform). And a generation unit 25. The printer-side controller 20 also includes a panel control unit 26 that controls the panel unit 15.

  When the memory card 6 is inserted into the slot 21, the printer-side controller 20 reads out the image file stored in the memory card 6 and stores it in the memory 23. Then, the printer-side controller 20 converts the image data of the image file into print data for printing by the printing mechanism 10, controls the printing mechanism 10 based on the printing data, and prints an image on paper. This series of operations is called “direct printing”.

  “Direct printing” is not only performed by inserting the memory card 6 into the slot 21, but also by connecting the digital still camera 2 and the printer 4 with a cable (not shown). The panel unit 15 is used for setting direct printing. The panel unit 15 is also used to display a confirmation screen during direct printing and to input confirmation (described later).

  Further, the printer 4 includes a scanner unit 80. The printer-side controller 20 includes a scanner control unit 27 that controls the scanner unit 80. The printer-side controller 20 controls the scanner unit 80 via the scanner unit 27, causes the scanner unit 80 to read a document, and acquires image data indicating an image of the document from the scanner unit 80.

=== Image File Structure ===
The image file is composed of image data and additional data. The image data is composed of a plurality of pixel data. The pixel data is data indicating pixel color information (gradation value). An image is formed by arranging the pixels in a matrix. Therefore, the image data is data indicating an image. The additional data includes data indicating the characteristics of image data, shooting data, thumbnail image data, and the like.

Hereinafter, a specific structure of the image file will be described.
FIG. 3 is an explanatory diagram of the structure of the image file. The left side of the figure shows the overall configuration of the image file, and the right side shows the configuration of the APP1 area.

  The image file starts with a marker indicating SOI (Start of image) and ends with a marker indicating EOI (End of Image). After the marker indicating SOI, there is an APP1 marker indicating the start of the data area of APP1. The APP1 data area after the APP1 marker includes additional data such as shooting data and thumbnail images. In addition, image data is included after the marker indicating SOS (Start of Stream).

  After the APP1 marker, there is information indicating the size of the data area of APP1, followed by an EXIF header and a TIFF header, which becomes an IFD area.

  Each IFD area has a plurality of directory entries, a link indicating the position of the next IFD area, and a data area. For example, in the first IFD0 (IFD of main image), the position of the next IFD1 (IFD of thumbnail image) is linked. However, since there is no IFD next to IFD 1 here, IFD 1 does not link to another IFD. Each directory entry includes a tag and a data portion. When the amount of data to be stored is small, the actual data is stored as it is in the data portion. When the amount of data is large, the actual data is stored in the IFD0 data area, and the data portion indicates the data storage location. A pointer is stored. The IFD 0 includes a directory entry in which a tag (Exif IFD Pointer) indicating the storage location of the Exif Sub IFD and a pointer (offset value) indicating the storage location of the Exif Sub IFD are stored.

  The Exif SubIFD area has a plurality of directory entries. This directory entry also includes a tag and a data portion. When the amount of data to be stored is small, the actual data is stored as it is in the data portion, and when the amount of data is large, the actual data is stored in the ExifSubIFD data area, and the data portion indicates the data storage location. A pointer is stored. In the ExifSubIFD, a tag indicating the storage location of the Makernote IFD and a pointer indicating the storage location of the Makernote IFD are stored.

  The Makernote IFD area has a plurality of directory entries. This directory entry also includes a tag and a data portion. When the amount of data to be stored is small, the actual data is stored as it is in the data portion. When the amount of data is large, the actual data is stored in the Makernote IFD data area, and the data portion stores the data. Is stored. However, since the data storage format can be freely defined in the Makernote IFD area, it is not always necessary to store data in this format. In the following description, data stored in the Makernote IFD area is referred to as “MakerNote data”.

FIG. 4A is an explanatory diagram of tags used in IFD0. As shown in the drawing, general data (data indicating the characteristics of image data) is stored in IFD0, and detailed photographing data is not stored.
FIG. 4B is an explanatory diagram of tags used in the Exif SubIFD. As shown in the figure, Exif SubIFD stores detailed shooting data. Note that most of the shooting data extracted in the scene identification process is shooting data stored in the Exif SubIFD. The shooting scene type tag (Scene Capture Type) is a tag indicating the type of shooting scene. Further, the Makernote tag is a tag indicating the storage location of the Makernote IFD.

  If the data portion (shooting scene type data) for the shooting scene type tag in the Exif SubIFD area is “zero”, it means “standard”, “1” means “landscape”, and “2” means “person”. "3" means "night view". Since the data stored in the Exif SubIFD is standardized, anyone can know the contents of the shooting scene type data.

  In the present embodiment, shooting mode data is included in one of the Makernote data. In this shooting mode data, a different value is shown for each mode set by the mode setting dial 2A. However, since the MakerNote data has a different format for each manufacturer, the contents of the shooting mode data cannot be known unless the format of the MakerNote data is known.

  FIG. 5 is a correspondence table between the setting of the mode setting dial 2A and data. Since the shooting scene type tag used in the Exif SubIFD conforms to the file format standard, the scenes that can be specified are limited, and data for specifying a scene such as “evening scene” cannot be stored in the data portion. On the other hand, since the MakerNote data can be freely defined, the data specifying the shooting mode of the mode setting dial 2A can be stored in the data portion by the shooting mode tag which is one of the MakerNote data.

  The above-described digital still camera 2 shoots under shooting conditions corresponding to the setting of the mode setting dial 2 </ b> A, then creates the above image file and stores it in the memory card 6. In this image file, shooting scene type data and shooting mode data corresponding to the mode setting dial 2A are stored in the Exif SubIFD area and the Makernote IFD area, respectively, as scene information added to the image data.

=== Outline of automatic correction function ===
When printing a “person” photo, there is a demand to clean the skin tone. In addition, when printing a “landscape” photograph, there is a demand for emphasizing the blue of the sky and the green of trees and grass. Therefore, the printer 4 of the present embodiment includes an automatic correction function that analyzes an image file and automatically performs a suitable correction process.

  FIG. 6 is an explanatory diagram of the automatic correction function of the printer 4. Each element of the printer-side controller 20 in the figure is realized by software and hardware.

  The storage unit 31 is realized by a partial area of the memory 23 and the CPU 22. All or part of the image file read from the memory card 6 is developed in the image storage unit 31 </ b> A of the storage unit 31. In addition, the calculation result of each element of the printer-side controller 20 is stored in the result storage unit 31B of the storage unit 31.

  The face identification unit 32 is realized by the CPU 22 and a face identification program stored in the memory 23. The face identification unit 32 analyzes the image data stored in the image storage unit 31A and identifies the presence or absence of a face. When the face identifying unit 32 identifies that there is a face, the image to be identified is identified as belonging to the “person” scene. Since the face identification process by the face identification unit 32 is the same as a process that has already been widely performed, detailed description thereof is omitted.

  The face identifying unit 32 also calculates the probability (certainty factor) that the image to be identified belongs to the scene of “person”. This certainty factor is calculated from the ratio of the skin color pixels in the image, the shape of the skin color image, the degree of proximity between the color indicated by the pixel data and the memory color of the skin color, and the like. The identification result of the face identification unit 32 is stored in the result storage unit 31B.

  The scene identification unit 33 is realized by the CPU 22 and a scene identification program stored in the memory 23. The scene identification unit 33 analyzes the image file stored in the image storage unit 31A and identifies the scene of the image indicated by the image data. After the face identifying process by the face identifying unit 32, the scene identifying process by the scene identifying unit 33 is performed. As will be described later, the scene identification unit 33 identifies whether the image to be identified is an image of “landscape”, “evening scene”, “night scene”, “flower”, or “autumn leaves”. Note that the identification result of the scene identification unit 33 and information on the certainty factor are also stored in the result storage unit 31B.

FIG. 7 is an explanatory diagram of a relationship between an image scene and correction contents.
The image correction unit 34 is realized by the CPU 22 and an image correction program stored in the memory 23. The image correction unit 34 converts the image data of the image storage unit 31A based on the identification results (identification results of the face identification unit 32 and the scene identification unit 33) stored in the result storage unit 31B (described later) of the storage unit 31. to correct. For example, when the identification result of the scene identification unit 33 is “landscape”, correction is performed so that blue is emphasized and green is emphasized. However, when the scene indicated by the additional data of the image file and the scene indicated by the identification result do not match, the image correction unit 34 corrects the image data according to the confirmation result after performing a predetermined confirmation process described later.

  The printer control unit 35 is realized by a printer control program stored in the CPU 22, the drive signal generation unit 25, the control unit 24, and the memory 23. The printer control unit 35 converts the corrected image data into print data, and causes the printing mechanism 10 to print the image.

=== Scene Identification Processing ===
FIG. 8 is a flowchart of scene identification processing by the scene identification unit 33. FIG. 9 is an explanatory diagram of the function of the scene identification unit 33. Each element of the scene identification unit 33 in the figure is realized by software and hardware. The scene identification unit 33 includes a feature amount acquisition unit 40, an overall classifier 50, a partial classifier 60, and an integrated classifier 70 shown in FIG.

First, the feature amount acquisition unit 40 analyzes the image data developed in the image storage unit 31A of the storage unit 31 and acquires partial feature amounts (S101). Specifically, the feature amount acquisition unit 40 divides the image data into 8 × 8 64 blocks, calculates the color average and variance of each block, and acquires the color average and variance as partial feature amounts. Here, each pixel has gradation value data in the YCC color space, and the average value of Y, the average value of Cb, and the average value of Cr are calculated for each block, and the variance of Y and the variance of Cb are calculated. And the variance of Cr are calculated respectively. That is, three color averages and three variances are calculated as partial feature amounts for each block. These color averages and variances indicate the characteristics of the partial images in each block. Note that an average value or variance in the RGB color space may be calculated.
Since the color average and variance are calculated for each block, the feature amount acquisition unit 40 expands the image data for the blocks in the block order without expanding all the image data in the image storage unit 31A. For this reason, the image storage unit 31A does not necessarily have a capacity sufficient to expand all of the image files.

  Next, the feature amount acquisition unit 40 acquires the entire feature amount (S102). Specifically, the feature quantity acquisition unit 40 acquires the overall color average, variance, center of gravity, and shooting information of the image data as the overall feature quantity. Note that these color averages and variances indicate the overall characteristics of the image. The color average, variance, and center of gravity of the entire image data are calculated using the partial feature values calculated previously. For this reason, it is not necessary to re-expand the image data when calculating the entire feature amount, and the calculation speed of the entire feature amount is increased. Although the overall identification process (described later) is performed prior to the partial identification process (described later), the overall feature value is obtained after the partial feature value in order to increase the calculation speed. The shooting information is extracted from the shooting data of the image file. Specifically, information such as the aperture value, shutter speed, and the presence or absence of flash emission is used as the overall feature amount. However, not all shooting data of the image file is used as the entire feature amount.

  Next, the overall classifier 50 performs overall identification processing (S103). The overall identification process is a process for identifying (estimating) an image scene indicated by image data based on the overall feature amount. Details of the overall identification process will be described later.

If the scene can be identified by the overall identification process (YES in S104), the scene identification unit 33 determines the scene by storing the identification result in the result storage unit 31B of the storage unit 31 (S109), and performs the scene identification process. finish. That is, when the scene can be identified by the overall identification process (YES in S104), the partial identification process and the integrated identification process are omitted. This increases the speed of the scene identification process.
If the scene cannot be identified by the overall identification process (NO in S104), the partial classifier 60 performs the partial identification process (S105). The partial identification process is a process for identifying the scene of the entire image indicated by the image data based on the partial feature amount. Details of the partial identification processing will be described later.

When the scene can be identified by the partial identification process (YES in S106), the scene identification unit 33 determines the scene by storing the identification result in the result storage unit 31B of the storage unit 31 (S109), and performs the scene identification process. finish. That is, when the scene can be identified by the partial identification process (YES in S106), the integrated identification process is omitted. This increases the speed of the scene identification process.
If the scene cannot be identified by the partial identification process (NO in S106), the integrated discriminator 70 performs the integrated identification process (S107). Details of the integrated identification process will be described later.

  When the scene can be identified by the integrated identification process (YES in S108), the scene identification unit 33 determines the scene by storing the identification result in the result storage unit 31B of the storage unit 31 (S109), and performs the scene identification process. finish. On the other hand, when the scene cannot be identified by the integrated identification process (NO in S108), the scene identification unit 33 stores all the candidate scenes (scene candidates) in the result storage unit 31B (S110). At this time, the certainty factor is stored in the result storage unit 31B together with the scene candidate.

  As a result of the scene identification process (whole identification process / partial identification process / integrated identification process), if any of S104, S106, and S108 in FIG. 8 is YES, the printer-side controller 20 selects one scene with a relatively high certainty factor. Can be identified. If NO in S108, the printer-side controller 20 can identify at least one scene (scene candidate) with a relatively low certainty factor. If NO in S108, there may be one scene candidate or two or more scene candidates.

=== Overall identification processing ===
FIG. 10 is a flowchart of the overall identification process. Here, the overall identification process will be described with reference to FIG.

  First, the overall classifier 50 selects one sub-classifier 51 from the plurality of sub-classifiers 51 (S201). The overall classifier 50 is provided with five sub-classifiers 51 for identifying whether an image to be identified (identification target image) belongs to a specific scene. The five sub classifiers 51 identify scenes of scenery, evening scene, night scene, flowers, and autumn leaves, respectively. Here, the overall classifier 50 selects the sub classifier 51 in the order of landscape → evening scene → night scene → flower → autumn leaves. For this reason, first, the sub classifier 51 (landscape classifier 51L) for identifying whether or not the classification target image belongs to a landscape scene is selected.

Next, the overall classifier 50 refers to the classification target table and determines whether or not a scene should be identified using the selected sub-classifier 51 (S202).
FIG. 11 is an explanatory diagram of the identification target table. This identification target table is stored in the result storage unit 31B of the storage unit 31. In the identification target table, all fields are set to zero in the first stage. In the process of S202, the “No” column is referred to, and if it is zero, it is determined as YES, and if it is 1, it is determined as NO. Here, the overall classifier 50 refers to the “No” column in the “Scenery” column in the identification target table, and determines “YES” because it is zero.

  Next, the sub classifier 51 calculates the value of the discriminant based on the entire feature amount (S203). The value of this discriminant is related to the probability (confidence) that the identification target image belongs to a specific scene (described later). For the sub classifier 51 of this embodiment, a classification method using a support vector machine (SVM) is used. The support vector machine will be described later. When the classification target image belongs to a specific scene, the discriminant of the sub classifier 51 tends to be a positive value. When the classification target image does not belong to a specific scene, the discriminant of the sub classifier 51 tends to be a negative value. Further, the discriminant becomes a larger value as the certainty that the identification target image belongs to the specific scene is higher. For this reason, if the discriminant value is large, the probability (confidence) that the identification target image belongs to a specific scene is high, and if the discriminant value is small, the probability that the classification target image belongs to a specific scene is low. .

  Next, the sub discriminator 51 determines whether or not the value of the discriminant is larger than the positive threshold (S204). If the value of the discriminant is larger than the positive threshold, the sub discriminator 51 determines that the classification target image belongs to a specific scene.

  FIG. 12 is an explanatory diagram of an affirmative threshold value of the overall identification process. In the figure, the horizontal axis indicates an affirmative threshold, and the vertical axis indicates the probability of recall or precision. FIG. 13 is an explanatory diagram of Recall and Precision. If the discriminant value is greater than or equal to the positive threshold, the identification result is Positive. If the discriminant value is not greater than or equal to the positive threshold, the identification result is Negative.

  Recall indicates the recall rate and detection rate. Recall is the ratio of the number of images identified as belonging to a specific scene to the total number of images of the specific scene. In other words, Recall is the probability that the sub-identifier 51 identifies the image as a positive when the image of the specific scene is identified by the sub-identifier 51 (the probability that the image of the specific scene belongs to the specific scene. ). For example, when a landscape image is identified by the landscape classifier 51L, it indicates the probability that the landscape classifier 51L identifies it as belonging to a landscape scene.

  Precision indicates the correct answer rate and the correct answer rate. Precision is the ratio of the number of images in a particular scene to the total number of images identified as Positive. In other words, Precision indicates the probability that the image to be identified is a specific scene when the sub-classifier 51 that identifies the specific scene identifies it as Positive. For example, when the landscape classifier 51L identifies that it belongs to a landscape scene, it indicates the probability that the identified image is really a landscape image.

As can be seen from FIG. 12, the larger the positive threshold, the greater the Precision. For this reason, the larger the positive threshold value, the higher the probability that an image identified as belonging to a landscape scene, for example, is a landscape image. That is, the greater the positive threshold, the lower the probability of misidentification.
On the other hand, the larger the positive threshold, the smaller the Recall. As a result, for example, even when a landscape image is identified by the landscape classifier 51L, it is difficult to correctly identify it as belonging to a landscape scene. By the way, if the image to be identified can be identified as belonging to a landscape scene (YES in S204), the speed of the overall identification process is increased so as not to identify other remaining scenes (such as sunsets). For this reason, the larger the positive threshold, the lower the overall identification processing speed. Further, if the scene can be identified by the overall identification process, the partial identification process is not performed and the speed of the scene identification process is increased (S104). Therefore, as the positive threshold is increased, the scene identification process speed decreases. Become.
That is, if the positive threshold is too small, the probability of misidentification increases, and if it is too large, the processing speed decreases. In the present embodiment, since the accuracy rate (Precision) is set to 97.5%, the landscape affirmation threshold is set to 1.72.

If the discriminant value is greater than the affirmative threshold value (YES in S204), the sub-classifier 51 determines that the classification target image belongs to a specific scene and sets an affirmative flag (S205). “Set an affirmative flag” means that the “affirmation” column in FIG. In this case, the overall discriminator 50 ends the overall discrimination process without performing discrimination by the next sub discriminator 51. For example, if the image can be identified as a landscape image, the entire identification process is terminated without identifying the sunset scene or the like. In this case, since the identification by the next sub-identifier 51 is omitted, the speed of the overall identification process can be increased.
If the value of the discriminant is not greater than the positive threshold (NO in S204), the sub discriminator 51 cannot determine that the classification target image belongs to a specific scene, and performs the next process of S206.

  Next, the sub discriminator 51 compares the discriminant value with a negative threshold value (S206). Thereby, the sub classifier 51 determines whether the classification target image does not belong to a predetermined scene. There are two types of such determinations. First, if the value of the discriminant of the sub-identifier 51 of a specific scene is smaller than the first negative threshold, it is determined that the classification target image does not belong to the specific scene. For example, if the discriminant value of the landscape classifier 51L is smaller than the first negative threshold, it is determined that the classification target image does not belong to a landscape scene. Second, if the value of the discriminant of the sub-identifier 51 of a specific scene is larger than the second negative threshold, it is determined that the classification target image does not belong to a scene different from the specific scene. . For example, if the discriminant value of the landscape classifier 51L is larger than the second negative threshold, it is determined that the classification target image does not belong to the night scene.

  FIG. 14 is an explanatory diagram of the first negative threshold. In the figure, the horizontal axis indicates the first negative threshold, and the vertical axis indicates the probability. The bold line in the graph is a True Negative Recall graph, and indicates the probability of correctly identifying an image other than a landscape image as not a landscape image. The thin line in the graph is a False Negative Recall graph, which indicates the probability of erroneously identifying a landscape image that is not a landscape image.

As can be seen from FIG. 14, the False Negative Recall is reduced as the first negative threshold is reduced. For this reason, the smaller the first negative threshold, the lower the probability that an image identified as not belonging to a landscape scene is a landscape image, for example. That is, the probability of misidentification is reduced.
On the other hand, the True Negative Recall decreases as the first negative threshold decreases. As a result, it is difficult to identify an image other than a landscape image unless it is a landscape image. On the other hand, if it is possible to identify that the identification target image is not a specific scene, the process by the sub partial classifier 61 for the specific scene is omitted during the partial identification process to speed up the scene identification processing speed (described later in FIG. 17). S302). For this reason, the scene identification processing speed decreases as the first negative threshold is decreased.
That is, if the first negative threshold is too large, the probability of misidentification increases, and if it is too small, the processing speed decreases. In this embodiment, in order to set False Negative Recall to 2.5%, the first negative threshold is set to −1.01.

  By the way, if the probability that an image belongs to a landscape scene is high, the probability that the image belongs to a night scene is inevitably low. For this reason, when the discriminant value of the landscape discriminator 51L is large, it may be identified that the scene is not a night scene. In order to perform such identification, a second negative threshold is provided.

  FIG. 15 is an explanatory diagram of the second negative threshold. In the figure, the horizontal axis indicates the value of the landscape discriminant, and the vertical axis indicates the probability. In this figure, the Recall graph of the night view is drawn with a dotted line together with the Recall and Precision graph of FIG. When attention is paid to this dotted line graph, if the value of the discriminant of landscape is larger than −0.44, the probability that the image is a night scene image is 2.5%. In other words, if the landscape discriminant value is greater than −0.44, even if the image is identified as not a night scene image, the probability of misidentification is only 2.5%. Therefore, in the present embodiment, the second negative threshold is set to −0.44.

  When the discriminant value is smaller than the first negative threshold value, or when the discriminant value is larger than the second negative threshold value (YES in S206), the sub-classifier 51 determines that the classification target image belongs to a predetermined scene. It is determined not to do so, and a negative flag is set (S207). “Set a negative flag” means to set the “No” column in FIG. For example, when it is determined that the image to be identified does not belong to a landscape scene based on the first negative threshold, the “denial” column in the “landscape” column is 1. Further, when it is determined that the identification target image does not belong to the night scene based on the second negative threshold, the “Negation” field in the “Night scene” field is “1”.

  FIG. 16A is an explanatory diagram of threshold values in the landscape classifier 51L described above. An affirmative threshold value and a negative threshold value are preset in the landscape discriminator 51L. 1.72 is set as the positive threshold. The negative threshold includes a first negative threshold and a second negative threshold. -1.01 is set as the first negative threshold. In addition, a value is set for each scene other than the landscape as the second negative threshold.

  FIG. 16B is an explanatory diagram outlining the processing of the landscape classifier 51L described above. Here, for simplification of description, only the night view will be described for the second negative threshold. If the discriminant value is greater than 1.72 (YES in S204), the landscape classifier 51L determines that the classification target image belongs to a landscape scene. If the discriminant value is 1.72 or less (NO in S204) and is greater than −0.44 (YES in S206), the landscape classifier 51L determines that the classification target image does not belong to the night scene. To do. If the value of the discriminant is smaller than −1.01 (YES in S206), the landscape classifier 51L determines that the classification target image does not belong to a landscape scene. Note that the landscape classifier 51L also determines whether the image to be identified does not belong to the scene based on the second negative threshold for the evening scene and the autumn leaves. However, since the second negative threshold for flowers is larger than the positive threshold, the landscape discriminator 51L does not determine that the classification target image does not belong to the flower scene.

  In the case of NO in S202, in the case of NO in S206, or when the processing in S207 is completed, the overall discriminator 50 determines the presence or absence of the next sub discriminator 51 (S208). Here, since the process by the landscape classifier 51L is finished, the overall classifier 50 determines in S208 that there is a next sub-classifier 51 (evening scene classifier 51S).

  Then, when the process of S205 is finished (when it is determined that the identification target image belongs to a specific scene), or when it is determined in S208 that there is no next sub-classifier 51 (the identification target image is a specific image). When it cannot be determined that the scene belongs to the scene), the overall discriminator 50 ends the overall discrimination process.

  As already described, when the overall identification process is completed, the scene identification unit 33 determines whether or not the scene has been identified by the overall identification process (S104 in FIG. 8). At this time, the scene identification unit 33 refers to the identification target table in FIG. 11 and determines whether or not there is 1 in the “affirmation” column.

  If the scene can be identified by the overall identification process (YES in S104), the partial identification process and the integrated identification process are omitted. This increases the speed of the scene identification process.

=== Partial identification processing ===
FIG. 17 is a flowchart of the partial identification process. The partial identification process is performed when the scene cannot be identified by the overall identification process (NO in S104 of FIG. 8). As will be described below, the partial identification process is a process of identifying the scene of the entire image by identifying each scene of the divided divided image. Here, the partial identification process will be described with reference to FIG.

  First, the partial classifier 60 selects one sub partial classifier 61 from the plurality of sub partial classifiers 61 (S301). The partial discriminator 60 is provided with three sub partial discriminators 61. Each sub partial discriminator 61 discriminates whether or not each partial image divided into 8 × 8 64 blocks belongs to a specific scene. Here, the three sub partial classifiers 61 identify the scenes of sunset, flowers, and autumn leaves, respectively. The partial discriminator 60 selects the sub partial discriminator 61 in the order of evening scene → flower → autumn leaves. Therefore, first, the sub partial classifier 61 (evening scene partial classifier 61S) for identifying whether or not the partial image belongs to the sunset scene is selected.

  Next, the partial discriminator 60 refers to the discrimination target table (FIG. 11) and determines whether or not the scene should be discriminated using the selected sub partial discriminator 61 (S302). Here, the partial discriminator 60 refers to the “No” column of the “Evening Scene” column in the classification target table, and determines YES if it is zero, and NO if it is 1. When the evening scene classifier 51S sets a negative flag with the first negative threshold during the overall identification process or when another sub-classifier 51 sets a negative flag with the second negative threshold, in S302 It is judged as NO. If it is determined NO, the sunset partial identification process is omitted, and the partial identification process speed increases. However, for the convenience of explanation, it is assumed that YES is determined here.

Next, the sub partial discriminator 61 selects one partial image from the partial images divided into 8 × 8 64 blocks (S303).
FIG. 18 is an explanatory diagram of the order of partial images selected by the evening scene partial classifier 61S. When a scene of the entire image is identified from the partial image, it is desirable that the partial image used for identification is a portion where the subject exists. Therefore, in this embodiment, thousands of samples of sunset scene images are prepared, each sunset scene image is divided into 64 blocks of 8 × 8, and blocks including sunset scene partial images (sun and sky partial images of the sunset scene) are included. The presence probability of the sunset partial image in each block was calculated based on the extracted block position. And in this embodiment, a partial image is selected in an order from a block with a high existence probability. Note that the selection order information shown in the figure is stored in the memory 23 as part of the program.

  In the case of an evening scene image, since the sky of the evening scene often spreads from the vicinity of the center to the upper half, the existence probability increases in the upper half block from the vicinity of the center. In the case of an evening scene image, the lower 1/3 of the image is shaded by backlight, and the partial image alone often cannot be distinguished from the evening scene or the night scene, so the existence probability is lower in the lower 1/3 block. In the case of a flower image, since the composition is often such that a flower is arranged near the center, the probability of existence of a flower partial image near the center increases.

  Next, the sub partial classifier 61 determines whether or not the partial image belongs to a specific scene based on the partial feature amount of the selected partial image (S304). Similar to the sub classifier 51 of the overall classifier 50, the sub partial classifier 61 uses a discrimination method using a support vector machine (SVM). The support vector machine will be described later. If the discriminant value is a positive value, it is determined that the partial image belongs to a specific scene, and the sub partial classifier 61 increments the positive count value. If the discriminant value is a negative value, it is determined that the partial image does not belong to a specific scene, and the sub partial discriminator 61 increments the negative count value.

  Next, the sub partial discriminator 61 determines whether or not the positive count value is larger than the positive threshold value (S305). The positive count value indicates the number of partial images determined to belong to a specific scene. If the positive count value is larger than the affirmative threshold (YES in S305), the sub partial classifier 61 determines that the classification target image belongs to a specific scene, and sets an affirmative flag (S306). In this case, the partial discriminator 60 ends the partial discriminating process without performing discrimination by the next sub partial discriminator 61. For example, if the image can be identified as an evening scene image, the partial identification process is terminated without identifying flowers and autumn leaves. In this case, since the identification by the next sub partial classifier 61 is omitted, the speed of the partial classification process can be increased.

  If the positive count value is not greater than the positive threshold value (NO in S305), the sub partial classifier 61 cannot determine that the classification target image belongs to a specific scene, and performs the next process of S307.

  If the sum of the positive count value and the number of remaining partial images is smaller than the positive threshold (YES in S307), the sub partial discriminator 61 proceeds to the process of S309. If the sum of the positive count value and the number of remaining partial images is smaller than the positive threshold value, the positive count value does not become larger than the positive threshold value even if the positive count value is incremented by all the remaining partial images. By proceeding with the process, the remaining partial images are not identified by the support vector machine. Thereby, the speed of the partial identification process can be increased.

If the sub partial discriminator 61 determines NO in S307, the sub partial discriminator 61 determines whether there is a next partial image (S308). In the present embodiment, not all of the partial images divided into 64 are selected in order. In FIG. 18, only the top 10 partial images indicated by thick frames are selected in order. Therefore, when the identification of the tenth partial image is completed, the sub partial classifier 61 determines in S308 that there is no next partial image. (In consideration of this point, the “number of remaining partial images” in S307 is also determined.)
FIG. 19 is a Recall and Precision graph when an evening scene image is identified using only the top 10 partial images. If an affirmative threshold as shown in the figure is set, the accuracy rate (Precision) can be set to about 80%, the recall rate (Recall) can be set to about 90%, and identification with high accuracy is possible.

In this embodiment, the evening scene image is identified using only 10 partial images. For this reason, in the present embodiment, it is possible to increase the speed of the partial identification process compared to the case where the evening scene image is identified using all 64 partial images.
In this embodiment, the sunset scene image is identified using the top tenth partial image having a high existence probability of the sunset scene partial image. For this reason, in the present embodiment, it is possible to set both Recall and Precision higher than the identification of an evening scene image using 10 partial images extracted by ignoring the existence probability.
In this embodiment, the partial images are selected in descending order of the existence probability of the sunset partial image. As a result, the determination in S305 is likely to be YES at an early stage. For this reason, in the present embodiment, the speed of the partial identification process can be increased as compared with the case where the partial images are selected in the order in which the presence probability level is ignored.

  When it is determined YES in S307, or when it is determined that there is no next partial image in S308, the sub partial discriminator 61 determines whether or not the negative count value is larger than the negative threshold (S309). . Since this negative threshold performs substantially the same function as the negative threshold (S206 in FIG. 10) in the above-described overall identification process, detailed description thereof is omitted. If YES is determined in S309, a negative flag is set as in S207 of FIG.

  In the case of NO in S302, in the case of NO in S309, or when the process of S310 is completed, the partial discriminator 60 determines whether or not there is a next sub partial discriminator 61 (S311). In the case after the processing by the evening scene partial classifier 61S is finished, since the flower partial classifier 61F and the autumnal leaves partial classifier 61R are still present as the sub partial classifier 61, the partial classifier 60 determines the next sub partial classifier in S311. It is determined that there is a container 61.

  Then, when the process of S306 is completed (when it is determined that the identification target image belongs to a specific scene), or when it is determined in S311 that there is no next sub partial classifier 61 (the identification target image is specified). If it cannot be determined that the scene belongs to the scene), the partial discriminator 60 ends the partial discrimination processing.

  As already described, when the partial identification process ends, the scene identification unit 33 determines whether or not the scene can be identified by the partial identification process (S106 in FIG. 8). At this time, the scene identification unit 33 refers to the identification target table in FIG. 11 and determines whether or not there is 1 in the “affirmation” column.

  When the scene can be identified by the partial identification process (YES in S106), the integrated identification process is omitted. This increases the speed of the scene identification process.

=== Support vector machine ===
Before describing the integrated identification process, the support vector machine (SVM) used in the sub-identifier 51 for the overall identification process and the sub-partial identifier 61 for the partial identification process will be described.

FIG. 20A is an explanatory diagram of determination by the linear support vector machine. Here, the learning sample is shown in a two-dimensional space by two feature amounts x1 and x2. The learning sample is divided into two classes A and B. In the figure, samples belonging to class A are indicated by circles, and samples belonging to class B are indicated by squares.
A boundary that divides the two-dimensional space into two is defined by learning using the learning sample. The boundary is defined by <w · x> + b = 0 (where x = (x1, x2), w is a weight vector, and <w · x> is an inner product of w and x). However, the boundary is defined by learning using a learning sample so that the margin is maximized. That is, in the case of the figure, the boundary is not a thick dotted line but a thick solid line.

The discrimination is performed using the discriminant f (x) = <w · x> + b. It is determined that a certain input x (this input x is different from the learning sample) belongs to class A if f (x)> 0, and belongs to class B if f (x) <0. Determined.
Here, the description is made using a two-dimensional space, but the present invention is not limited to this (that is, the feature amount may be two or more). In this case, the boundary is defined by a hyperplane.

  By the way, there are cases where the two classes cannot be separated by a linear function. In such a case, if the determination is performed by the linear support vector machine, the accuracy of the determination result is lowered. Therefore, if the feature quantity of the input space is nonlinearly transformed, that is, if the input space is nonlinearly mapped to a certain feature space, it can be separated by a linear function in the feature space. This is used in the nonlinear support vector machine.

  FIG. 20B is an explanatory diagram of discrimination using a kernel function. Here, the learning sample is shown in a two-dimensional space by two feature amounts x1 and x2. If the nonlinear mapping from the input space of FIG. 20B becomes a feature space as shown in FIG. 20A, it can be separated into two classes by a linear function. If the boundary is defined so that the margin is maximized in this feature space, the inverse mapping of the boundary in the feature space becomes the boundary shown in FIG. 20B. As a result, the boundary becomes nonlinear as shown in FIG. 20B.

In this embodiment, by using a Gaussian kernel, the discriminant f (x) becomes as follows (where M is the number of features and N is the number of learning samples (or contributes to the boundary): The number of learning samples), w _i is a weighting factor, y _j is the feature quantity of the learning sample, and x _j is the feature quantity of the input x).

  It is determined that a certain input x (this input x is different from the learning sample) belongs to class A if f (x)> 0, and belongs to class B if f (x) <0. Determined. Further, the larger the value of the discriminant f (x), the higher the probability that the input x (this input x is different from the learning sample) belongs to the class A. On the contrary, the smaller the value of the discriminant f (x), the lower the probability that the input x (this input x is different from the learning sample) belongs to the class A. In the sub-identifier 51 for the overall identification process and the sub-partial identifier 61 for the partial identification process, the value of the discriminant f (x) of the support vector machine is used.

  An evaluation sample is prepared separately from the learning sample. The above Recall and Precision graphs are based on the identification results for the evaluation samples.

=== Integrated identification processing ===
In the above-described overall identification process and partial identification process, the positive threshold value in the sub-classifier 51 and the sub-classifier 61 is set relatively high, and the Precision (correct answer rate) is set high. This is because, for example, if the accuracy rate of the landscape classifier 51L of the overall classifying unit is set low, the landscape classifier 51L misidentifies the autumnal image as a landscape image, and before the autumnal classifier 51R performs classification. This is because a situation occurs in which the entire identification process ends. In the present embodiment, by setting the Precision (accuracy rate) high, an image belonging to a specific scene is identified by the sub-classifier 51 (or sub-partial classifier 61) of the specific scene ( For example, the autumnal leaves image is identified by the autumnal leaves discriminator 51R (or the autumnal leaf partial discriminator 61R).

  However, if the Precision (accuracy rate) of the overall identification process or the partial identification process is set to be high, there is a high possibility that the scene cannot be identified by the overall identification process or the partial identification process. Therefore, in this embodiment, when the scene cannot be identified by the overall identification process and the partial identification process, the integrated identification process described below is performed.

  FIG. 21 is a flowchart of the integrated identification process. As will be described below, the integrated identification process is based on the discriminant value of each sub-classifier 51 in the overall identification process, and the certainty level is equal to or higher than a predetermined value (for example, 90% or higher). This is a process for selecting a high scene.

  First, the integrated discriminator 70 extracts a positive scene based on the discriminant values of the five sub discriminators 51 (S401). At this time, the value of the discriminant calculated by each sub classifier 51 during the overall identification process is used.

  Next, the integrated discriminator 70 determines whether or not there is a scene with a certainty factor equal to or greater than a predetermined value (S402). Here, the certainty factor indicates the probability that the identification target image belongs to the specific scene, and is obtained from the value of the discriminant. More specifically, the integrated discriminator 70 is provided with a table showing the relationship between the discriminant value and the precision, and a precision corresponding to the discriminant value is derived based on this table. The value is used as confidence. The predetermined value is set to 90%, for example, and is set to a value lower than Precision (97.5%) set by an affirmative threshold of the overall classifier or partial classifier. However, the certainty factor may not be Precision, and the value of the discriminant may be used as the certainty factor.

  If there is a scene with a certainty level equal to or greater than a predetermined value (YES in S402), an affirmative flag is set in the field for that scene (S403), and the integrated identification process is terminated. Note that when a scene having a certainty factor of 90% or more is extracted, a plurality of scenes are not extracted. This is because if the certainty of a certain scene is high, the certainty of another scene is inevitably low.

  On the other hand, if there is no scene with the certainty level equal to or greater than the predetermined value (NO in S402), the integrated identification process is terminated without setting an affirmative flag. As a result, one scene does not exist in the affirmative column of the identification target table in FIG. That is, it cannot be identified to which scene the identification target image belongs.

  As already described, when the integrated identification process is completed, the scene identification unit 33 determines whether or not the scene has been identified by the integrated identification process (S108 in FIG. 8). At this time, the scene identification unit 33 refers to the identification target table in FIG. 11 and determines whether or not there is 1 in the “affirmation” column. If YES is determined in S402, the determination in S108 is also YES. On the other hand, if NO is determined in S402, the determination in S108 is also NO.

  In the present embodiment, in the case of NO in S108 in FIG. 8, that is, in the case of NO in S402 in FIG. 21, all the scenes extracted in S401 are stored as scene candidates in the result storage unit 31B.

=== First Embodiment ===
<Overview>
As described above, the user can set the shooting mode with the mode setting dial 2A. The digital still camera 2 determines shooting conditions (exposure time, ISO sensitivity, etc.) based on the set shooting mode, photometry results at the time of shooting, and the like, and shoots the subject under the determined shooting conditions. After shooting, the digital still camera 2 stores shooting data indicating shooting conditions at the time of shooting together with the image data in the memory card 6 as an image file.

  By the way, since the user forgot to set the shooting mode, shooting may be performed while the shooting mode inappropriate for the shooting condition is set. For example, a daytime landscape may be shot while the night view mode is set. In this case, although the image data of the image file is an image of a daytime landscape, data indicating the night view mode is stored in the shooting data (for example, the shooting scene type data in FIG. 5 is “3”). "become). In such a case, if the image data is corrected based on inappropriate shooting scene type data, printing with an image quality undesirable for the user is performed.

  On the other hand, even if the image data is corrected based on the result of the identification processing (face identification processing and scene identification processing), printing with the image quality desired by the user may not be obtained. For example, when an erroneous identification occurs in the identification process, there is a case where a print having an image quality desired by the user cannot be obtained. Further, even when the shooting mode set by the user for a special effect is denied and the image data is corrected based on the identification result by the printer, printing as intended by the user cannot be performed.

  Therefore, in this embodiment, a confirmation screen for prompting the user to confirm is displayed. Specifically, as described later, when the identification processing result and the scene indicated by the scene information (shooting scene type data and shooting mode data) of the additional data of the image file do not match, the display unit 16 of the panel unit 15 A confirmation screen is displayed.

<Explanation of confirmation screen display>
FIG. 22 is a flowchart showing the flow of direct print processing in the first embodiment. Each process is realized by the printer-side controller 20 based on a program stored in the memory 23.
First, the printer-side controller 20 performs face identification processing by the face identification unit 32 and scene identification processing by the scene identification unit 33 (S501). Since these processes have already been described, a description thereof will be omitted.

  Next, the printer-side controller 20 determines whether the scene indicated by the additional data can be compared with the scene indicated by the identification processing result (S502). When a plurality of scene candidates are included in the identification processing result, determination is made using the scene candidate with the highest certainty factor.

Note that the determination method in S502 is different between the case where shooting scene type data is used for determination of mismatch in the next S503 and the case where shooting mode data which is MakerNote data is used.
When shooting scene type data is used in S503, when the shooting scene type data is neither “person”, “landscape”, or “night view”, for example, when shooting scene type data is “0” (see FIG. 5), in S503. Since the comparison with the identification processing result cannot be performed, the determination in S502 is NO. If the identification processing result is neither “person”, “landscape”, or “night view”, the comparison with the photographic scene type data cannot be made in S503, and therefore the determination in S502 is NO. For example, when the identification processing result is “evening scene”, the determination in S502 is NO.
When shooting mode data is used in S503, when the shooting mode data is not “person”, “landscape”, “evening scene”, or “night scene”, for example, when the shooting mode data is “3 (close-up)” (see FIG. 5). Since the comparison with the identification processing result cannot be performed, the determination in S502 is NO. When the identification processing result is neither “person”, “landscape”, “evening scene”, or “night scene”, the comparison with the shooting mode data is impossible, and the determination in S502 is NO.

  In the case of YES in S502, the printer-side controller 20 determines whether or not the scene indicated by the additional data (shooting scene type data and shooting mode data) and the scene indicated by the identification processing result do not match (S503). When a plurality of scene candidates are included in the identification processing result, determination is made using the scene candidate with the highest certainty factor.

  If they do not match (YES in S503), the printer-side controller 20 displays a confirmation screen on the display unit 16 (S504). This confirmation screen is used to indicate which scene the image data is to be corrected based on when the scene indicated by the additional data (shooting scene type data and shooting mode data) and the scene indicated by the identification processing result do not match. This is a screen to let you confirm.

  FIG. 23 is an explanatory diagram of a comparative example of a confirmation screen. On the confirmation screen of this comparative example, the scene indicated by the additional data and the scene as the identification processing result are shown. However, on such a confirmation screen, when correction processing is performed based on each scene, it is difficult for the user to recognize which image (which correction processing) has the desired image quality.

  FIG. 24 is an explanatory diagram of a first example of a confirmation screen. On the confirmation screen 161, two images indicated by the image data are displayed side by side. The image 161A on the left side is an image corrected based on the scene indicated by the additional data, and is corrected in the “landscape” mode here (see FIG. 7). On the other hand, the image 161B on the right side is an image corrected based on the scene of the identification processing result, and is corrected in the “person” mode here (see FIG. 7). On the confirmation screen 161, since two images corrected in different correction modes are displayed side by side, the user can easily compare the effects of the correction processing. Also, the certainty of the identification processing result is shown below the right image. Since the image displayed on the display unit 16 is not large, the user's judgment material is supplemented by displaying the certainty factor together with the image. The printer-side controller 20 highlights the edge of the image selected by the user with a bold line in response to the user's operation on the input unit 17. In the initial setting, the right image 161B is selected. If the OK button 161C is selected by the user (YES in S505), the printer-side controller 20 corrects the image data in the correction mode according to the user's selection (S506). If the OK button 161C is pressed in the state of FIG. 24, the image data is corrected in the “person” mode, which is the identification processing result.

  FIG. 25 is an explanatory diagram of a second example of the confirmation screen. On the confirmation screen 162, one image indicated by the image data is displayed. However, the left and right image quality are different at the boundary 162C. On the left side of the boundary, an image 162A corrected based on the scene indicated by the additional data is displayed. On the right side of the boundary, an image 162B corrected based on the scene of the identification processing result is displayed. By displaying in this way, the user can compare the left and right image quality at the boundary, so that it is easier to compare the two images (162A, 162B) than comparing the two separated images as shown in FIG. . Normally, the printer-side controller 20 sets the boundary 162C at the center of the screen, but when the identification processing result is “person”, the printer-side controller 20 sets the boundary at the center of the identified person as shown in the figure. To do. When the image is corrected in the “person” mode, the skin color is corrected (see FIG. 7). Therefore, by setting a boundary at the center of the person, the user can use the “person” mode. It can be easily determined whether or not image correction is appropriate. On this confirmation screen, a button 162D for displaying the scene indicated by the additional data and a button 162E for displaying the scene of the identification processing result are displayed. When one of the two buttons (162D, 162E) is selected by the user (YES in S505), the printer-side controller 20 corrects the image data in a correction mode according to the user's selection (S506).

  26A and 26B are explanatory diagrams of a third example of the confirmation screen. First, the confirmation screen of FIG. 26A is displayed on the display unit 16. On this confirmation screen 163, an image corrected based on the scene of the identification processing result is displayed. Here, correction in the “person” mode is performed (see FIG. 7). When the user selects the OK button 163A (YES in S505), the printer-side controller 20 corrects the image data in the correction mode according to the user's selection (S506). If the OK button 163A is pressed in the state of FIG. 26A, the image data is corrected in the “person” mode that is the identification processing result. A switching button 163B is displayed beside the OK button 163A. If the switching button 163B is pressed in the state of FIG. 26A, the confirmation screen of FIG. 26B is displayed on the display unit 16. On the confirmation screen 163, an image corrected based on the scene indicated by the additional data is displayed. Here, correction in the “landscape” mode is performed (see FIG. 7). If the OK button 163A is pressed in the state of FIG. 26B, the image data is corrected in the “person” mode that is the identification processing result. When the switch button 163B is pressed in the state of FIG. 26B, the confirmation screen of FIG. 26A is displayed on the display unit 16. According to the confirmation screen 163, since the displayed image is larger than the confirmation screen 161 of the first example, the user can easily compare the effects of the correction processing. Further, the user can compare two images in the entire range of the images by switching and displaying the two images corrected in different correction modes, so that the effect of the correction process can be easily compared. In particular, when the user attaches importance to the effect of correction processing in a specific portion (for example, sharpness in the eyes of a person who is a subject), this confirmation screen 163 is more convenient than the confirmation screen 162 of the second example. .

  The confirmation screen is displayed until the user inputs confirmation (NO in S505). If there is a confirmation input by the user (YES in S505), the printer-side controller 20 corrects the image data in the correction mode according to the user's selection (S506). However, even if there is no input by the user, if a predetermined time (for example, 20 seconds) elapses after the confirmation screen is displayed, it corresponds to the scene selected in the initial setting (here, the scene of the identification processing result). The image data may be corrected in the correction mode. Thereby, even if the user is away from the printer, the process can proceed. The scene selected in the initial setting may be changed according to the certainty of the identification processing result, or may be determined in advance so that the scene of the identification processing result is always selected regardless of the certainty.

  If NO in S502, the comparison cannot be made in S503, and the process proceeds to S506. If NO in S503, confirmation with the user is unnecessary, and the process proceeds to S506.

  In step S506, the printer-side controller 20 corrects the image data in a predetermined correction mode. If the user confirms in step S505, the printer-side controller 20 corrects the image data based on the scene selected by the user. If NO in S502 or S503, the printer-side controller 20 corrects the image data based on the scene of the identification processing result.

  After the image correction process, the printer-side controller 20 prints an image based on the corrected image data (S507). Thereby, a print image with an appropriate image quality can be obtained.

=== Second Embodiment ===
In the first embodiment described above, the printer-side controller 20 displays a confirmation screen on the display unit 16 in order to prompt the user to confirm. In contrast, in the second embodiment, the user's confirmation is prompted by printing an order sheet.

  FIG. 27 is a flowchart showing the flow of direct print processing in the second embodiment. Each process is realized by the printer-side controller 20 based on a program stored in the memory 23. FIG. 28A to FIG. 28C are explanatory diagrams of the state of direct printing according to the second embodiment. As shown in FIG. 28A, when the memory card 6 is inserted into the slot 21, the printer-side controller 20 starts direct print processing.

  Of the processes in FIG. 27, the processes of S501 to S503 performed first are the same as those in the first embodiment described above, and thus the description thereof is omitted.

  If they do not match (YES in S503), the printer-side controller 20 controls the printing mechanism 10 to cause the printing mechanism 10 to print the order sheet (S511, see FIG. 28B). In this order sheet, when the scene indicated by the additional data (shooting scene type data and shooting mode data) and the scene indicated by the identification processing result do not match, the user determines which scene the image data is to be corrected based on. It is a printed matter to make you confirm.

FIG. 29 is an explanatory diagram of an order sheet.
On the order sheet 9, two images indicated by the image data are printed side by side. The image 901 on the left is an image corrected based on the scene indicated by the additional data, and is corrected in the “landscape” mode here (see FIG. 7). On the other hand, the image 902 on the right side is an image that has been corrected based on the scene of the identification processing result, and is corrected in the “person” mode here (see FIG. 7). In this order sheet, since two images corrected in different correction modes are printed side by side, the user can easily compare the effects of the correction processing. In addition, since the effect of the correction process can be confirmed based on the actually printed image as compared with the case where the image is evaluated by the display unit 16 that is a liquid crystal display, the user can obtain the printed image according to the desire. Cheap. In addition, the certainty of the identification processing result is also shown below the right image 902. Since the image printed on the order sheet 9 is not large, the user's judgment material is supplemented by printing the certainty factor together with the image.

  An ellipse mark for instructing the number of prints is printed below each image. The user can instruct the printer 4 to select which of the two correction processes by painting an ellipse mark under the image that has been subjected to the correction process according to the user's desire. For example, when instructing the printer 4 to print one image corrected in the “person” mode, the user may fill in the “1” ellipse mark under the right image 902. It is also possible to instruct printing of both images.

  Note that an alignment mark 911 is also printed on the order sheet 9. The alignment mark 911 indicates the orientation of the order sheet 9 when the order sheet 9 is placed on the platen glass of the scanner unit 80 of the printer 4. That is, when the user places the order sheet 9 on the platen glass of the scanner unit 80 of the printer 4 (see FIG. 28C), the order sheet is set so that the alignment mark 911 faces the origin position of the platen glass. 9 is set in the scanner unit 80. The alignment mark 911 is also used for skew detection when the order sheet 9 is placed obliquely on the scanner unit 80.

An attribute mark 912 indicating the attribute of the order sheet is also printed on the order sheet 9. The attributes of the order sheet include the total number of prints of the order sheet, the page number of the order sheet, the order sheet type, and the like. The attribute mark 912 is used for skew detection together with the alignment mark 911.
Further, the user can select the paper type in the paper type selection area 913 of the order sheet 9. Further, in the border presence / absence selection area 914 of the order sheet 9, the user performs borderless printing in which an image is printed without creating a margin on the edge of the printing paper, or the image is printed with a margin on the edge of the printing paper. It is possible to select whether to perform borderless printing.

  In this way, the user can instruct the printer of desired printing conditions by selectively painting the oval mark printed on the order sheet 9 with a pencil. After filling in the order sheet 9, the user sets the order sheet 9 on the platen glass of the scanner unit 80 as shown in FIG. 28C. Then, after setting the order sheet 9 to the scanner unit 80, the user operates the input unit 17 of the panel unit 15 (for example, by pressing a scanner button) to cause the scanner unit 80 to read the order sheet 9 (in S512). YES).

  The printer-side controller 20 acquires the image data of the order sheet 9 from the scanner unit 80, analyzes the image data, and identifies the mark filled by the user (S513). Then, the printer-side controller 20 corrects the image data in the correction mode according to the user's selection according to the user's filled mark (S506). For example, if the printer-side controller 20 identifies that the oval mark “1” under the right image 902 is filled based on the image data obtained by reading the order sheet, the printer-side controller 20 becomes a target for direct printing. The image data of the image file is corrected in the “person” mode (see FIG. 7).

  If NO in S502, the comparison cannot be made in S503, and the process proceeds to S506. If NO in S503, confirmation with the user is unnecessary, and the process proceeds to S506. If NO in S502 or S503, the printer-side controller 20 corrects the image data based on the scene of the identification processing result (S506).

  After the image correction process, the printer-side controller 20 prints an image based on the corrected image data (S507). Thereby, a print image with an appropriate image quality can be obtained.

=== Add scene information to additional data ===
If the user selects a scene on the confirmation screen (or order sheet), the scene desired by the user can be determined. Therefore, in this embodiment, when the user selects on the confirmation screen (or order sheet), the scene selected by the user is stored in the additional data of the image file. Here, a case where the user selects an image that has been corrected based on the scene of the identification processing result on the confirmation screen will be described.

FIG. 30 is an explanatory diagram of the configuration of the APP1 area when the identification result is added to the additional data. In FIG. 30, a different part compared with the image file of FIG. 3 is shown by a thick line.
Compared with the image file of FIG. 3, Makernote IFD is added to the image file of FIG. Information of the identification processing result is stored in the second Makernote IFD.

A new directory entry is also added to the Exif SubIFD. The added directory entry is composed of a tag indicating the second Makernote IFD and a pointer indicating the storage location of the second Makernote IFD.
In addition, since a new directory entry is added to the Exif Sub IFD, the storage location of the Exif Sub IFD data area is shifted, so the pointer indicating the storage location of the Exif Sub IFD data area is changed.
Further, since the second Makernote IFD is added, the IFD1 area is shifted, so that the link indicating the position of IFD1 is also changed in IFD0. In addition, since the second Makernote IFD is added, the size of the data area of APP1 is changed, so the size of the data area of APP1 is also changed.

  Thus, by saving the user-selected scene (in this case, the scene of the identification processing result) in the additional data of the image file, when the image of this image file is printed again, the identification processing and the confirmation screen are displayed. It becomes unnecessary. Further, when the user takes out the memory card 6 from the printer 4 of the present embodiment and inserts the memory card 6 into another printer, this printer does not have a scene identification processing function but performs automatic correction processing. However, the image data can be corrected appropriately.

=== Other Embodiments ===
Although a printer or the like as one embodiment has been described, the above embodiment is for facilitating understanding of the present invention, and is not intended to limit the present invention. The present invention can be changed and improved without departing from the gist thereof, and it is needless to say that the present invention includes equivalents thereof. In particular, the embodiments described below are also included in the present invention.

<About the printer>
In the above-described embodiment, the printer 4 performs scene identification processing, confirmation screen display, and the like. However, the digital still camera 2 may perform scene identification processing, confirmation screen display, and the like. Further, the information processing apparatus that performs the scene identification process and the confirmation screen display is not limited to the printer 4 or the digital still camera 2. For example, an information processing apparatus such as a photo storage that stores a large amount of image files may perform the above-described scene identification processing and confirmation screen display. Of course, a personal computer or a server installed on the Internet may perform the above-described scene identification processing and confirmation screen display.

<About image files>
The image file described above is in the Exif format, but the image file format is not limited to this. Further, the above-described image file is a still image, but may be a moving image. In short, if the image file includes image data and additional data, the scene identification process as described above can be performed.

<About Support Vector Machine>
For the above-described sub classifier 51 and sub partial classifier 61, a classification method using a support vector machine (SVM) is used. However, the method for identifying whether or not the identification target image belongs to a specific scene is not limited to using a support vector machine. For example, pattern recognition such as a neural network may be employed.

<About scene candidate extraction method>
In the above-described embodiment, a scene having a certainty level equal to or higher than a predetermined value is extracted as a scene candidate only when a scene cannot be identified by any of the overall identification process, the partial identification process, and the integrated identification process. However, the method for extracting scene candidates is not limited to this.
FIG. 31 is an explanatory diagram of another processing flow. This process is performed instead of the scene identification process described above.

First, as in the above-described embodiment, the printer-side controller 20 calculates an overall feature amount based on image file information (S601). Next, the landscape discriminator 51L calculates the discriminant value or the Precision corresponding to the discriminant value as the certainty factor as in the above-described discrimination processing (S602). Note that the landscape classifier 51L of the above-described embodiment also identifies whether or not the classification target image belongs to a landscape scene. However, the landscape classifier 51L here has a certainty factor based on a discriminant. Just calculate. Similarly, the other sub classifiers 51 also calculate the certainty factor (S603 to S606). Then, the printer-side controller 20 extracts scenes having a certainty factor equal to or greater than a predetermined value as scene candidates (S607), and stores the scene candidates (and certainty factors) (S608).
Even in this way, it is possible to identify the scene of the image indicated by the image data. It is also possible to compare the scene identified in this way with the scene of the additional data, and display a confirmation screen or print an order sheet when there is a mismatch.

=== Summary ===
(1) In the above-described embodiment, the printer-side controller 20 acquires shooting scene type data and shooting mode data that are scene information from the additional data added to the image data. Further, the printer-side controller 20 acquires the identification result obtained by the face identification process or the scene identification process (see FIG. 8).
There are cases where the scene indicated by the shooting scene type data or shooting mode data does not match the scene identified by the face identification process or the scene identification process. Therefore, in the above-described embodiment, when two scenes do not match, a confirmation screen is displayed or an order sheet is printed to prompt the user to confirm. However, as shown in the confirmation screen of the comparative example in FIG. 23, if an image subjected to correction processing based on each scene is not displayed, the user determines which image (which correction processing) has the desired image quality. It is difficult to recognize.

  Therefore, in the above-described embodiment, when the two scenes do not match, the printer-side controller 20 indicates to the user the image corrected based on the scene indicated by the additional data and the image corrected based on the scene of the identification result. I have decided. As a result, the user can easily compare the effects of the correction process, and the user's confirmation work is simplified.

(2) According to the first embodiment described above, when the two scenes do not match, the printer-side controller 20 prompts the user to confirm by displaying a confirmation screen on the display unit 16. On this confirmation screen, an image corrected based on the scene indicated by the additional data and an image corrected based on the scene of the identification result are displayed. Thereby, the user can confirm the effect of the correction process on the display unit 16.

(3) According to the confirmation screen 161 (see FIG. 24) of the first example of the first embodiment, two images indicated by the image data are displayed side by side. On the confirmation screen 161, since two images corrected in different correction modes are displayed side by side, the user can easily compare the effects of the correction processing.

(4) On the confirmation screen 162 (see FIG. 25) of the second example of the first embodiment, one image indicated by the image data is displayed. At that time, the printer-side controller 20 sets a boundary at a predetermined position of the image, displays an image corrected based on the scene of the additional data on the left side of the boundary, and displays the identification processing result on the right side of the boundary. An image corrected based on the scene is displayed. By displaying in this way, the user can compare the left and right image quality at the boundary, so that it is easier to compare two images than to compare two separate images as shown in FIG.

(5) When the identification processing result is “person”, the identification target image includes a human face. On the other hand, when the image is corrected in the “person” mode, since the skin color is corrected (see FIG. 7), the correction effect tends to appear in the face portion of the person. Therefore, when the identification processing result is “person”, the printer-side controller 20 sets a boundary 162C with the face portion as the center, as in the above-described confirmation screen 162 in FIG. Thereby, it can be easily determined whether or not the correction is appropriate.
Note that setting the position of the boundary according to the position of the subject in this way is not necessarily limited to the case where the identification processing result is “person”. For example, if the identification processing result is “landscape”, the image data is corrected in the landscape mode if the boundary position is set so as to pass the blue or green portion (memory color portion of the landscape mode) of the identification target image This makes it easier to check the effect of correction.

(6) According to the confirmation screen 163 (FIGS. 26A and 26B) of the third example of the first embodiment, the printer-side controller 20 corrects an image based on the scene indicated by the additional data according to the user's instruction. And an image corrected based on the scene of the identification result. Thereby, since the displayed image can be enlarged, the user can easily compare the effects of the correction processing. Further, the user can compare two images in the entire range of the images by switching and displaying the two images corrected in different correction modes, so that the effect of the correction process can be easily compared.

(7) According to the second embodiment, when two scenes do not match, an image corrected based on the scene indicated by the additional data and an image corrected based on the scene of the identification result are printed on paper. An order sheet for prompting the user to confirm is created. Compared with the case where the image is evaluated by the display unit 16 which is a liquid crystal display, in the second embodiment, the effect of the correction process can be confirmed based on the actually printed image. Easy to get.

(8) The above-described printer 4 (corresponding to an information processing apparatus) includes a printer-side controller 20 (see FIG. 2). The printer-side controller 20 acquires shooting scene type data and shooting mode data as scene information from the additional data added to the image data. Further, the printer-side controller 20 acquires the identification result obtained by the face identification process or the scene identification process (see FIG. 8). When the scene indicated by the shooting scene type data or the shooting mode data does not match the scene identified by the scene identification process, the printer controller 20 corrects the image corrected based on the scene indicated by the additional data and the identification result. An image corrected based on the scene is shown to the user. As a result, the user can easily compare the effects of the correction process, and the user's confirmation work is simplified.

(9) The above-described memory 23 stores a program that causes the printer 4 to execute the processes of FIGS. 22 and 27. That is, the program includes a code for acquiring scene information of image data from additional data added to image data, a code for identifying an image scene indicated by image data based on the image data, and a scene indicated by scene information. If the scene identified by the scene information and the identified scene do not match, the image corrected based on the scene indicated by the scene information and the image corrected based on the identified scene And a code shown to the user.

It is explanatory drawing of an image processing system. 2 is an explanatory diagram of a configuration of a printer. FIG. It is explanatory drawing of the structure of an image file. FIG. 4A is an explanatory diagram of tags used in IFD0. FIG. 4B is an explanatory diagram of tags used in the Exif SubIFD. It is a correspondence table of mode setting dial settings and data. It is explanatory drawing of the automatic correction function of a printer. It is explanatory drawing of the relationship between the scene of an image, and the correction content. It is a flowchart of the scene identification process by a scene identification part. It is explanatory drawing of the function of a scene identification part. It is a flowchart of a whole identification process. It is explanatory drawing of an identification object table. It is explanatory drawing of the affirmation threshold value of the whole identification process. It is explanatory drawing of Recall and Precision. It is explanatory drawing of a 1st negative threshold value. It is explanatory drawing of a 2nd negative threshold value. FIG. 16A is an explanatory diagram of threshold values in the landscape classifier. FIG. 16B is an explanatory diagram of an outline of the process of the landscape classifier. It is a flowchart of a partial identification process. It is explanatory drawing of the order of the partial image which an evening scene partial identifier selects. It is a Recall and Precision graph when the evening scene image is identified only by the top 10 partial images. FIG. 20A is an explanatory diagram of determination by the linear support vector machine. FIG. 20B is an explanatory diagram of discrimination using a kernel function. It is a flowchart of an integrated identification process. It is a flowchart which shows the flow of the direct printing process in 1st Embodiment. It is explanatory drawing of the comparative example of a confirmation screen. It is explanatory drawing of the 1st example of a confirmation screen. It is explanatory drawing of the 2nd example of a confirmation screen. 26A and 26B are explanatory diagrams of a third example of the confirmation screen. It is a flowchart which shows the flow of the process of the direct print in 2nd Embodiment. It is explanatory drawing of the mode of the direct printing of 2nd Embodiment. It is explanatory drawing of an order sheet. It is explanatory drawing of a structure of an APP1 area | region when an identification result is added to additional data. It is explanatory drawing of another processing flow.

Explanation of symbols

2 Digital still camera, 2A mode setting dial,
4 Printer, 6 Memory card, 9 Order sheet,
10 printing mechanism, 11 head, 12 head control unit, 13 motor, 14 sensor, 15 panel unit, 16 display unit, 17 input unit,
20 printer-side controller, 21 slots, 22 CPU, 23 memory,
24 control unit, 25 drive signal generator,
31 storage unit, 31A image storage unit, 31B result storage unit,
32 face identification unit, 33 scene identification unit, 34 image correction unit, 35 printer control unit,
40 feature quantity acquisition unit, 50 global classifier, 51 sub classifier, 51L landscape classifier,
51S evening scene classifier, 51N night scene classifier, 51F flower classifier, 51R autumn leaves classifier,
60 partial classifiers, 61 sub partial classifiers, 61S evening scene partial classifiers,
61F Flower partial classifier, 61R Autumn colored partial classifier,
70 Integrated identifier,
161/162/163 confirmation screen, 162C border 901/902 image, 911 alignment mark, 912 attribute mark,
913 Paper type selection area, 914 Edge presence / absence selection area

Claims (9)

From the additional data added to the image data, obtain the scene information of the image data,
Based on the image data, the scene of the image indicated by the image data is identified,
Comparing the scene indicated by the scene information with the identified scene;
When the scene indicated by the scene information and the identified scene do not match, information indicating the image corrected based on the scene indicated by the scene information and the image corrected based on the identified scene is displayed to the user Processing method. An information processing method according to claim 1,
An information processing method, comprising: displaying a confirmation screen on a display unit when showing the corrected image to a user. An information processing method according to claim 2,
An information processing method, wherein an image corrected based on a scene indicated by the scene information and an image corrected based on an identified scene are displayed side by side on the confirmation screen. An information processing method according to claim 2,
While displaying the image indicated by the image data on the confirmation screen,
Setting a boundary at a predetermined position of the image indicated by the image data;
One side viewed from the boundary is an image corrected based on the scene indicated by the scene information,
An information processing method characterized in that an image corrected based on an identified scene is set on the other side as viewed from the boundary. An information processing method according to claim 4,
When the identified scene is a scene indicating that there is a human face in the image,
The information processing method, wherein the boundary is set so as to pass through the face portion. An information processing method according to claim 2,
One image of the image corrected based on the scene indicated by the scene information and the image corrected based on the identified scene is displayed on the confirmation screen, and switched to the other image according to a user instruction. An information processing method characterized by displaying. An information processing method according to claim 1,
An information processing method comprising: printing an image corrected based on a scene indicated by the scene information and an image corrected based on an identified scene on a medium when the corrected image is shown to a user. An information processing apparatus comprising a controller,
The controller is
From the additional data added to the image data, obtain the scene information of the image data,
Based on the image data, the scene of the image indicated by the image data is identified,
Comparing the scene indicated by the scene information with the identified scene;
When the scene indicated by the scene information and the identified scene do not match, information indicating the image corrected based on the scene indicated by the scene information and the image corrected based on the identified scene is displayed to the user Processing equipment. In the information processing device,
From the additional data added to the image data, the scene information of the image data is acquired,
Based on the image data, the scene of the image indicated by the image data is identified,
Comparing the scene indicated by the scene information with the identified scene;
When the scene indicated by the scene information and the identified scene do not match, the user is caused to indicate an image corrected based on the scene indicated by the scene information and an image corrected based on the identified scene. program.