JP5381498B2 - Image processing apparatus, image processing program, and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing program, and an image processing method.

  A technique for discriminating the semantic classification (for example, a shooting scene) of each image based on the feature amount of the image is known (see Patent Document 1). In general, data used for image discrimination is calculated using a support vector machine (SVM) method based on a feature amount of a sample image, and is called a discriminator. A plurality of discriminators are also used for one event (such as an event or an event) (see Patent Document 2).

US Pat. No. 7,035,467 JP 2005-100121 A

  Each event determination using a plurality of discriminators has a problem that the burden of the determination process increases.

  An image processing apparatus according to the present invention includes a calculation unit that calculates a feature amount of an event determination target image, a determination unit that performs event determination on the event determination target image by comparing the calculated feature amount with an event determination classifier, When there are a plurality of classifiers for the same event, a determination unit for determining one representative classifier based on characteristic data indicating characteristics for each classifier, and using the determined representative classifier Control means for controlling the determination means.

  According to the present invention, the burden of event determination processing can be reduced.

It is a block diagram explaining the principal part structure of the electronic camera by one embodiment of this invention. It is a flowchart explaining the flow of the event determination process which main CPU performs. It is a flowchart explaining the detail of an event determination process. It is a flowchart explaining the detail of a discriminator determination process. It is a figure which illustrates image number accuracy data. It is a flowchart explaining the detail of a learning process. It is a flowchart explaining the detail of the preparation process of discriminator characteristic data. It is a flowchart explaining the discriminator determination process by the modification 4. It is a flowchart explaining the discriminator determination process by the modification 5. It is a figure which illustrates a computer apparatus.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. FIG. 1 is a block diagram for explaining a main configuration of an electronic camera 1 according to an embodiment of the present invention. The electronic camera 1 is controlled by the main CPU 11.

  The photographing lens 21 forms a subject image on the imaging surface of the imaging element 22. The imaging element 22 is configured by a CCD image sensor or the like, captures a subject image on the imaging surface, and outputs an imaging signal to the imaging circuit 23. R (red), G (green), and B (blue) color filters are respectively provided on the imaging surface of the imaging element 22 so as to correspond to the pixel positions. Since the image sensor 22 captures a subject image through the color filter, the photoelectric conversion signal output from the image sensor 22 has RGB color system color information.

  The imaging circuit 23 performs analog processing (such as gain control) on the photoelectric conversion signal output from the imaging element 22, and converts the analog imaging signal into digital data by a built-in A / D conversion circuit.

  The main CPU 11 inputs a signal output from each block, performs a predetermined calculation, and outputs a control signal based on the calculation result to each block. The image processing circuit 12 is configured as an ASIC, for example, and performs image processing on the digital image signal input from the imaging circuit 23. The image processing includes, for example, contour enhancement, color temperature adjustment (white balance adjustment) processing, and format conversion processing for an image signal.

  The image compression circuit 13 performs image compression processing at a predetermined compression ratio on the image signal processed by the image processing circuit 12 by, for example, the JPEG method. The display image creation circuit 15 creates display data for displaying the captured image on the liquid crystal monitor 16.

  The buffer memory 14 temporarily stores data before and after image processing and during image processing, stores an image file before being recorded on the recording medium 30, and stores an image file read from the recording medium 30. Used for.

  The recording medium 30 includes a memory card that can be attached to and detached from the electronic camera 1. The recording medium 30 records captured image data and an image file including the information in accordance with an instruction from the main CPU 11. The image file recorded on the recording medium 30 can be read by an instruction from the main CPU 11.

  The flash memory 19 stores a program executed by the main CPU 11, data necessary for processing performed by the main CPU 11, and the like. The contents of programs and data stored in the flash memory 19 can be added and changed by an instruction from the main CPU 11.

  The operation member 17 includes various buttons and switches of the electronic camera 1, and outputs an operation signal corresponding to the operation content of each operation member, such as a release button pressing operation and a mode switching switch switching operation, to the main CPU 11.

  The GPS device 18 receives radio waves from GPS satellites in response to instructions from the main CPU 11 and outputs received signals to the main CPU 11. The main CPU 11 performs a predetermined calculation based on the received signal from the GPS device 18 and detects the positioning information (latitude, longitude, altitude) of the electronic camera 1.

  The electronic camera 1 performs predetermined image processing and compression processing on the image signal acquired by the image sensor 22 at the time of shooting, and adds additional information including positioning information and information about the shot image to the image data after the compression processing. It is configured to generate an added image file. Specifically, an image file in Exif format is generated in which image data in JPEG format is stored in the image data portion and additional information is stored in the additional information portion. The Exif format image file is obtained by embedding thumbnail images and additional information data in image data in the JPEG image format. The generated image file is stored in the recording medium 30.

  Further, the electronic camera 1 is configured to be able to switch between a shooting mode and a playback mode. The shooting mode is an operation mode in which a subject image is shot and data of the shot image is stored in the recording medium 30 as an image file. The reproduction mode is a mode in which the reproduced image based on the image data is displayed on the liquid crystal monitor 16 by reading the captured image data from the recording medium 30.

<Event determination of captured images>
The electronic camera 1 of this embodiment has a function of grouping captured images. Specifically, event determination is performed on an image file recorded on the recording medium 30, and the image file after determination is stored in a folder provided for each event. Even if the image file is not stored in the folder provided for each event, an image file management table may be created and event information for the image file may be stored in the table.

  FIG. 2 is a flowchart for explaining the flow of event determination processing executed by the main CPU 11. When the operation signal instructing execution of the event determination process is input from the operation member 17, the main CPU 11 starts the process illustrated in FIG. 2.

  In step S10 of FIG. 2, the main CPU 11 determines whether or not to learn. When the learning instruction is input from the operation member 17, the main CPU 11 makes a positive determination in step S10 and proceeds to step S40. When the learning instruction is not input from the operation member 17, the main CPU 11 makes a negative determination in step S10 and proceeds to step S20.

  In step S20, the main CPU 11 determines whether or not to determine an event. When the event determination instruction is input from the operation member 17, the main CPU 11 makes a positive determination in step S20 and proceeds to step S30. When the event determination instruction is not input from the operation member 17, the main CPU 11 makes a negative determination in step S <b> 20 and ends the process of FIG. 2.

  In step S30, the main CPU 11 performs an event determination process, and ends the process of FIG. Details of the event determination process will be described later. In step S40 that proceeds with a negative determination in step S10 described above, the main CPU 11 performs a learning process and ends the process of FIG. Details of the learning process will be described later.

<Event determination processing>
Details of the event determination process (S30) for an image will be described with reference to the flowchart illustrated in FIG. For example, the main CPU 11 reads the image file data to be subject to event determination from the recording medium 30 and develops it in the buffer memory 14 to activate the processing shown in FIG. In step S31 of FIG. 3, the main CPU 11 calculates the feature amount of the image from one image and proceeds to step S32. The feature amount to be calculated is, for example, color information, sharpness information, texture information, pattern information, brightness information, and the like calculated based on pixel data that constitutes a predetermined area of the image. Further, image size and color histogram information may be handled as feature amounts. Further, the number of “faces” included in the image, age estimated from “faces”, and gender recognition information may be included in the feature amount. Since the feature amount calculation is a known technique, a detailed description of feature amount calculation is omitted in this description.

  In step S32, the main CPU 11 calculates the probability (score) representing the likelihood of each event using the discriminators created in advance for each event and recorded in the flash memory 19, and the process proceeds to step S33. The score representing the event likeness is a boundary that partitions the space in the feature space represented by the corresponding event identifier (for example, a boundary between an area corresponding to the event “athletic meet” and an area corresponding to the “non-athletic meet”). This corresponds to the distance from the feature amount calculated in step S31. The feature amount calculated from a certain image is located behind the feature amount region corresponding to “Athletic meet” in the feature amount space represented by the event identifier for “athletic meet”, and the feature amount region corresponding to “non-athletic meet” If the distance to is long, the score of “Athletic meet” is high. On the other hand, if the feature amount calculated from the image is located at the end of the feature amount region corresponding to “athletic meet” and the distance to the feature amount region corresponding to “non-athletic meet” is short, the score of “athletic meet” likelihood is low . The main CPU 11 calculates a score according to the distance.

  In step S33, the main CPU 11 determines whether or not there are a plurality of discriminators for the same event. When there is another discriminator for the same event in the flash memory 19, the main CPU 11 makes a positive determination in step S33 and proceeds to step S36. When there is no other discriminator for the same event in the flash memory 19, the main CPU 11 makes a negative determination in step S33 and proceeds to step S34.

  In step S34, the main CPU 11 determines whether or not processing has been completed for all event determination target images. The main CPU 11 makes a positive determination in step S34 and proceeds to step S35 when the feature amount calculation and the event likelihood (score) P are calculated for all event determination images. The main CPU 11 makes a negative determination in step S34 and returns to step S31 when there is an event determination image for which feature amount calculation and event likelihood (score) P are not calculated. When returning to step S31, the process with respect to another event determination image is repeated.

  In step S35, the main CPU 11 determines that the determination target image corresponds to the event corresponding to the best score, and ends the process of FIG.

  In step S36, which proceeds after making an affirmative decision in step S33, the main CPU 11 determines a discriminator to be used for event judgment from among a plurality of discriminators for the same event, and proceeds to step S34. In the determination of the discriminator, the discriminator having a high determination accuracy is determined by comparing the superiority of the discriminator using discriminator characteristic data described later.

  The details of the discriminator determination process will be described with reference to the flowchart illustrated in FIG. In step S36A of FIG. 4, the main CPU 11 performs an operation for comparing (N1 × P1) and (N2 × P2), and proceeds to step S36B. For example, P1 is a score calculated using the event identifier 1 for “athletic day”, and P2 is a score calculated using the event identifier 2 for “athletic day”, for example. N1 is the image number accuracy data of the event identifier 1, and N2 is the image number accuracy data of the event identifier 2. The image number accuracy data is one of the characteristic data of the discriminator, and is data indicating the relationship between the number of images used for generation (that is, learning) of the discriminator and the event determination accuracy by the discriminator. FIG. 5 is a diagram illustrating image number accuracy data. The main CPU 11 calculates image number accuracy data at the time of creating discriminator characteristic data in a learning process (FIG. 6) described later, and records it in the flash memory 19.

  In step S36B, the main CPU 11 determines whether or not (N1 × P1)> (N2 × P2) is satisfied. The main CPU 11 makes an affirmative decision in step S36B when (N1 × P1)> (N2 × P2) is satisfied, and proceeds to step S36C. If (N1 × P1)> (N2 × P2) is not satisfied, the main CPU 11 performs a step. A negative determination is made in S36B, and the process proceeds to step S36D.

  In step S36C, the main CPU 11 selects the event discriminator 1 and ends the process shown in FIG. In step S36D, the main CPU 11 selects the event discriminator 2 and ends the process shown in FIG.

  Note that in step S36A described above, the determination may be made simply based on the magnitude relationship between N1 and N2. In this case, the larger one of N1 and N2, that is, the event discriminator having the higher event determination accuracy is selected.

<Learning process>
The event discriminator described above is generated by learning based on sample image (learning image) data. The details of the learning process (S40) will be described with reference to the flowchart illustrated in FIG. The main CPU 11 inputs information indicating the storage location (for example, memory address) of the learning image and the event (for example, athletic meet) of the image, and starts the processing of FIG.

In step S41 of FIG. 6 , the main CPU 11 calculates a feature amount from one learning image. The feature quantity to be calculated is a feature quantity that matches the information indicating the input event, and corresponds to, for example, an event identifier for “athletic meet”. The feature amount is calculated in the same manner as in step S31 in FIG.

  In step S42, the main CPU 11 holds characteristic information indicating the shooting time, shooting location, and the like of the image from the additional information data of the learning image file, and proceeds to step S43.

  In step S43, the main CPU 11 determines whether or not the feature amount calculation and the characteristic information holding have been completed for all the learning images developed in the buffer memory 14. When the main CPU 11 has processed all the learning images, the main CPU 11 makes an affirmative decision in step S43 and proceeds to step S44. If the processing for all the learning images has not been completed, the main CPU 11 makes a negative determination in step S43 and returns to step S41. When returning to step S41, the above-described processing is repeated for the other learning images.

  In step S44, the main CPU 11 creates a discriminator. Specifically, a discriminator is created based on the feature amount of an image belonging to an event and the feature amount of an image not belonging to the event using an SVM (Support Vector Machine) method, and is stored in the flash memory 19. Proceed to step S45.

  In step S45, the main CPU 11 creates discriminator characteristic data and ends the processing shown in FIG. Details of the process of creating the discriminator characteristic data will be described with reference to the flowchart illustrated in FIG.

  In step S45A of FIG. 7, the main CPU 11 calculates the image number accuracy data of the discriminator from the number of images of learning images, and proceeds to step S45B. The image number accuracy data has the characteristics illustrated in FIG. In general, the event determination accuracy is saturated when the number of learning images used to create the discriminator reaches a predetermined number (for example, 500), and thereafter the event determination accuracy is substantially constant even when the number of images is increased. Have a common tendency to become.

  Image number accuracy data is calculated for each event classifier. The calculated image number accuracy data is recorded in the flash memory 19. The image number accuracy data may be expressed as a function expression, or may be a histogram or table data. In the present embodiment, when the number of learning images used for creating the event discriminator reaches a predetermined number (for example, 10), the event discriminator is enabled (can be used for event determination). Event discriminators in which the number of learning images is less than a predetermined number are recorded in the flash memory 19 but are not used for event determination. When the number of learning images used in the learning process reaches a predetermined number, the event discriminator is validated and allowed to be used at the time of subsequent event determination.

  In step S45B, the main CPU 11 uses the characteristic information indicating the shooting time recorded in the flash memory 19 in step S42 to calculate information (for example, a histogram) indicating the shooting date / time distribution, and ends the process of FIG.

According to the embodiment described above, the following effects can be obtained.
(1) The main CPU 11 of the electronic camera 1 calculates the feature amount of the event determination target image, performs event determination on the event determination target image by comparing the calculated feature amount with the event determination discriminator, and sets the same event. On the other hand, when there are a plurality of discriminators, one representative discriminator is determined based on the characteristic data indicating the characteristics of each discriminator, and the determined representative discriminator is controlled. Accordingly, for example, the processing load can be reduced as compared with a case where a plurality of determination processes are performed with a plurality of discriminators for the same event.

(2) The main CPU 11 further calculates a feature amount of the learning image, generates a discriminator for event determination based on the feature amount of the learning image, and indicates characteristics of the generated event determination discriminator The data was acquired. Accordingly, it is easy to additionally generate a new classifier or to learn by adding a new feature amount to an existing classifier. Learning an existing classifier leads to an improvement in event determination accuracy by the classifier.

(3) When the characteristic data includes information indicating the determination accuracy for each classifier and the representative classifier is determined based on the high level of determination accuracy, there are a plurality of classifiers for the same event. In addition, it is possible to make a determination by a better classifier.

(4) Since the classifier having the number of learning images exceeding the predetermined number is made valid and the representative classifier is determined from the validated classifiers, the determination is made with a small number of learning images. If there is a concern about insufficient accuracy, the risk of insufficient determination accuracy can be resolved by using another classifier other than the classifier.

(Modification 1)
In FIG. 7, the example in which both the image number accuracy data and the shooting date / time distribution data are calculated as the discriminator characteristic data has been described. However, either one may be calculated. In addition to the above two, profile information indicating the color characteristics of the image may be calculated. Furthermore, information indicating the distribution of shooting locations may be calculated using the characteristic information indicating the shooting locations recorded in the flash memory 19 in step S42 (FIG. 6).

(Modification 2)
When the electronic camera 1 is configured to be able to communicate with an external device, information related to the learning image may be acquired from the external device. For example, weather information corresponding to the shooting date and time is acquired and recorded in the flash memory 19 using information on the shooting date and time of the learning image. Thereby, the weather distribution information at the time of photographing can be added to the discriminator characteristic data.

(Modification 3)
When the electronic camera 1 is configured to be able to communicate between networks via an external device, the event identifier is shared between users who share a common hobby and preference, and the event identifier is learned by multiple users. You may make it make it.

(Modification 4)
The classifier determination process may be performed as follows. FIG. 8 is a flowchart for explaining a classifier determination process according to the fourth modification. In step S36E of FIG. 8, the main CPU 11 extracts information indicating the shooting date and time of the image from the additional information data of the event determination target image file, and proceeds to step S36F.

  In step S36F, the main CPU 11 determines whether or not acquisition of shooting date / time information has been completed for all event determination images. When the main CPU 11 has processed all the event determination images, the main CPU 11 makes a positive determination in step S36F and proceeds to step S36G. If the processing for all the event determination images has not been completed, the main CPU 11 makes a negative determination in step S36F and returns to step S36E. When returning to step S36E, the above-described processing is repeated for other event determination images.

  In step S <b> 36 </ b> G, the main CPU 11 calculates information (for example, a histogram) indicating the shooting date / time distribution of the event determination image. The main CPU 11 further calculates a logical product of areas (that is, the number of images having the same shooting date / time) with the shooting date / time distribution (for example, histogram) related to the “athletic event” event identifier 1 calculated in step S45B. Calculate and proceed to step S36H. Here, (calculated area) / (distribution area of the event determination image) is K1. K1 represents the number of images taken on the date and time when the shooting frequency is high in the learning image of the event discriminator 1 among the images of the event determination images.

  In step S36H, the main CPU 11 performs a logical AND of areas (that is, images having the same shooting date and time) with the shooting date and time distribution (for example, histogram) related to the “athletic event” event identifier 2 calculated in step S45B. Number) is calculated and the process proceeds to step S36A. Here, (calculated area) / (distribution area of the event determination image) is K2. K2 represents the number of images taken on the date and time when the shooting frequency is high in the learning image of the event discriminator 2 among the images of the event determination images.

  In step S36I, the main CPU 11 performs an operation for comparing (K1 × P1) and (K2 × P2), and proceeds to step S36J. K1 and K2 are as described above. P1 is a score calculated using the event discriminator 1, and P2 is a score calculated using the event discriminator 2.

  In step S36J, the main CPU 11 determines whether or not (K1 × P1)> (K2 × P2) is satisfied. The main CPU 11 makes an affirmative decision in step S36J when (K1 × P1)> (K2 × P2) is satisfied, and proceeds to step S36C. If (K1 × P1)> (K2 × P2) is not satisfied, the main CPU 11 performs a step. A negative determination is made in S36J, and the process proceeds to step S36D.

  In step S36C, the main CPU 11 selects the event discriminator 1 and ends the process shown in FIG. In step S36D, the main CPU 11 selects the event discriminator 2 and ends the process shown in FIG.

  In step S36J, the determination may be made simply based on the magnitude relationship between K1 and K2. In this case, the larger of K1 and K2, that is, the common image when the photographing date and time of the learning image used for generating the event identifier is the same as the photographing date and time of the event determination image. Choose the event identifier with the larger number. For example, since an “athletic meet” is usually held in the daytime, it is possible to select a discriminator that is generated using a learning image that is frequently photographed in the daytime period.

  Further, the average of (K1 × P1) and (K2 × P2) may be regarded as the final “athletic meet”, and the event likelihood (score) may be similarly calculated for other events to perform comparison processing. .

(Modification 5)
The classifier determination process may be performed as follows. FIG. 9 is a flowchart for explaining a classifier determination process according to the fifth modification. Compared with the flowchart of FIG. 8, steps S36K and S36L are different, and the difference will be mainly described.

  In FIG. 9, the main CPU 11 proceeds to step S36K after step S36H. The main CPU 11 performs an operation for comparing {(N1 × C + K1 × D) × P1} and {(N2 × C + K2 × D) × P2} and proceeds to step S36L. N1 is the image number accuracy data of the event identifier 1, and N2 is the image number accuracy data of the event identifier 2. C is the weight of the event identifier 1, and D is the weight of the event identifier 2. Each of these weighting data is included in the characteristic data of the event identifier. K1 and K2 are as described above. P1 is a score calculated using the event discriminator 1, and P2 is a score calculated using the event discriminator 2.

  In step S36L, the main CPU 11 determines whether {(N1 × C + K1 × D) × P1} is larger than {(N2 × C + K2 × D) × P2}. When {(N1 × C + K1 × D) × P1}> {(N2 × C + K2 × D) × P2} holds, the main CPU 11 makes an affirmative decision in step S36L and proceeds to step S36C, and {(N1 × C + K1 × If D) × P1}> {(N2 × C + K2 × D) × P2} is not satisfied, a negative determination is made in step S36L to proceed to step S36D.

  In step S36C, the main CPU 11 selects the event discriminator 1 and ends the process shown in FIG. In step S36D, the main CPU 11 selects the event discriminator 2 and ends the process shown in FIG.

  In step S36K, it may be determined that the larger discriminator is selected based on the magnitude relationship between (N1 × C + K1 × D) and (N2 × C + K2 × D).

(Modification 6)
In the above description, an example in which images are grouped in the electronic camera 1 has been described. However, the electronic camera 1 may be replaced with an electronic device such as a photo frame or a photo viewer.

(Modification 7)
The image processing apparatus may be configured by causing the computer apparatus 10 illustrated in FIG. 10 to execute an image processing program for performing the processes illustrated in FIGS. When an image processing program is taken into the personal computer 10 and used, the program is loaded into a data storage device of the personal computer 10 and then used as an image processing device for grouping images by executing the program.

  The loading of the program to the personal computer 10 may be performed by setting a recording medium 104 such as a CD-ROM storing the program in the personal computer 10 or by a method via the communication line 101 such as a network. You may load. When passing through the communication line 101, the program is stored in the hard disk device 103 of the server (computer) 102 connected to the communication line 101. The image processing program can be supplied as various types of computer program products such as provision via the recording medium 104 or the communication line 101.

  The above description is merely an example, and is not limited to the configuration of the above embodiment.

DESCRIPTION OF SYMBOLS 1 ... Electronic camera 10 ... Computer 11 ... Main CPU
14 ... Buffer memory 17 ... Operation member 19 ... Flash memory 30 ... Recording medium

Claims (7)

Calculating means for calculating the feature amount of the event determination target image;
A determination unit that performs event determination on the event determination target image by comparing the calculated feature amount with an event determination classifier;
Determining means for determining one representative classifier based on characteristic data indicating characteristics of each classifier when there are a plurality of the classifiers for the same event;
Control means for controlling the determination means to use the determined representative classifier;
An image processing apparatus comprising: The image processing apparatus according to claim 1 .
Before Kitoku property data includes information indicating the accuracy of determining each of the discriminator,
It said determining means, an image processing apparatus characterized by determining the representative discriminator based on the height of the determination accuracy. The image processing apparatus according to claim 1 or 2 ,
A second calculating means for calculating a feature quantity of academic習用image,
Generating means for generating a discriminator for event determination based on a feature amount of the learning image ;
And characteristic data acquiring means for acquiring characteristic data indicating characteristics of the identifier before Kisei made event determination,
An image processing apparatus further comprising: The image processing apparatus according to claim 3 .
Before Kitoku property data includes photographing date and time information of the learning image used for the generation of the identifier,
The image processing apparatus according to claim 1, wherein the determining unit determines the representative classifier based on a frequency of capturing the learning image. The image processing apparatus according to claim 3 or 4 ,
The determination means determines the representative classifier from the classifiers in which the number of learning images exceeds a predetermined number. A calculation process for calculating the feature amount of the event determination target image;
A determination process for determining one representative classifier based on characteristic data indicating characteristics for each classifier when there are a plurality of classifiers for determining an event for the same event;
A determination process for performing event determination on the event determination target image by comparing the calculated feature amount with the determined representative classifier;
An image processing program for causing a computer to execute. Calculate the feature amount of the event determination target image,
When there are a plurality of discriminators for event determination for the same event, one representative discriminator is determined based on the characteristic data indicating the characteristic of each discriminator,
An image processing method comprising: performing event determination on the event determination target image by comparing the calculated feature amount with the determined representative classifier.

Images