JP2017117405A - Image processing unit, program and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing unit capable of improving reproductivity of a layout result.SOLUTION: An image processing unit is configured to: obtain information on a specific portion, for instance, face information, from among a plurality of images being candidates of a layout object; determine an execution order for specific processing using the information, for instance, individual recognition processing; and according to the determined execution order, execute individual recognition processing.SELECTED DRAWING: Figure 11

Description

  The present invention relates to an image processing apparatus, a program, and an image processing method, and more particularly, to an image processing apparatus, a program, and an image processing method for creating a photo album from a plurality of images taken with a digital camera or the like.

  In recent years, a service for creating an album by automatically laying out images taken by a user has started to be provided. Here, automatic layout refers to automatic processing of some or all of the functions related to album creation. For example, (1) an image is automatically selected for album publication, (2) a template to be used for an album is automatically determined, and (3) an image is automatically assigned to a template.

  When the selected image is an image including a person, it is preferable that the image includes an important person. Many users may want to create an album with a “photograph that they have taken” rather than a “photo that the passer-by has taken” even if the composition and background of the person are the same Is expensive. There is a personal recognition technology as a technology necessary for making such a determination automatically.

  For example, Patent Document 1 discloses a configuration in which a dictionary is newly created, and collation processing and registration processing are sequentially performed on detected face information. Patent Document 1 describes that a dictionary that is optimal for each user is formed by creating a new dictionary in this way.

JP 2014-44606 A

  However, when the contents of the face dictionary are sequentially updated as in Patent Document 1, the “reproducibility” of the personal recognition result may not be maintained. For example, when face detection is performed in multiple threads, the start / end timing of each thread varies depending on the task usage status. As a result, even if the personal recognition process itself is operated by only one thread, the personal recognition result changes if the input order of the face information input to the single thread varies.

  Changing the personal recognition result leads to changing the automatic layout result. When creating an album using the same image many times, it is not preferable for the user to create an album with a different layout result each time.

  The present invention has been made in view of the above conventional example, and an object thereof is to provide an image processing apparatus, an image processing method, and a program capable of realizing the same layout configuration for the same image.

  In order to achieve the above object, the image processing apparatus of the present invention includes the following arrangement.

  That is, an image processing apparatus that lays out a plurality of image data, wherein an acquisition unit that acquires a plurality of image data, and information on a specific portion of each of the plurality of image data acquired by the acquisition unit are processed by multithread processing. Based on the sorting result obtained by sorting the information of the specific part of each of the plurality of image data detected by the detection means and the detection means detected by the detection means, the specific part of each of the plurality of image data A determination unit that determines an execution order of a specific process using information of the specific part for information; an execution unit that executes the specific process according to the execution order determined by the determination unit; and Layout means for laying out the plurality of image data based on a result of the executed specific processing. To.

  In another aspect of the present invention, a program for causing a computer to function as each unit of the apparatus having the above-described configuration is provided.

  Further, according to another aspect of the present invention, there is provided an image processing method in an image processing apparatus that lays out a plurality of image data, an acquisition step of acquiring a plurality of image data, and a plurality of acquired image data respectively. The plurality of images based on a detection step of detecting information on a specific part by multi-thread processing and a sorting result obtained by sorting information on the specific part of each of the detected plurality of image data A determination step of determining an execution order of a specific process using the information of the specific part with respect to information of the specific part of each data; an execution step of executing the specific process according to the determined execution order; A layout step of laying out the plurality of image data based on a result of the specific processing executed in the execution step. And an image processing method comprising and.

  Therefore, according to the present invention, when a specific process is executed using information obtained by detecting a specific part of each image, the same layout configuration can be obtained by using the same image data. .

It is a block diagram which shows the structure of the hardware of an image processing apparatus. It is a block diagram which shows the software structure of an album creation application. It is a figure which shows the display screen which an album creation application provides. It is a flowchart which shows an automatic layout process. It is a figure which shows the image analysis information distinguished for every ID. It is a figure which shows the result of the scene division | segmentation of an image data group. It is a figure which shows the average value and standard deviation for every scene. It is a figure which shows the concept of a scoring axis. It is a figure for demonstrating selection of image data. It is a figure which shows the template group used for the layout of image data. 3 is a block diagram illustrating a detailed configuration of a person information analysis unit 205. FIG. It is a figure which shows the example of a data table. It is a flowchart which shows the face detection process which the person information analysis part according to Embodiment 1 performs. It is a flowchart which shows the personal recognition process which the person information analysis part according to Embodiment 1 performs. It is a block diagram which shows the detailed structure of an image score part. It is a flowchart which shows the scoring process after the scene classification | category which an image scoring part performs. It is a figure which shows the selected template. 12 is a flowchart showing processing executed by an automatic layout processing unit of the album creating application according to the second embodiment. 10 is a flowchart for executing processing in units of pages according to the second embodiment in multi-threads. It is a figure which shows the example of an album of 2 pages of spreads. 10 is a flowchart showing details of album replacement processing according to the second embodiment. It is a flowchart which shows the face detection process which the person information analysis part according to Embodiment 3 performs. It is a flowchart which shows the personal recognition process which the person information analysis part according to Embodiment 3 performs.

  Hereinafter, embodiments of the present invention will be described more specifically and in detail with reference to the accompanying drawings. However, the relative arrangement and the like of the constituent elements described in this embodiment are not intended to limit the scope of the present invention only to those unless otherwise specified.

  In this specification, “recording” (sometimes referred to as “printing”) is not limited to the case of forming significant information such as characters and graphics, but may be significant. Furthermore, it also represents a case where an image, a pattern, a pattern, or the like is widely formed on a recording medium or a medium is processed regardless of whether or not it is manifested so that a human can perceive it visually.

  “Recording medium” refers not only to paper used in general recording apparatuses but also widely to cloth, plastic film, metal plate, glass, ceramics, wood, leather, and the like that can accept ink. Shall.

<Embodiment 1>
In this embodiment, an example in which an album creation application (hereinafter referred to as an application) is operated on an image processing apparatus to generate an automatic layout will be described. This embodiment is characterized in that the face detection process can be executed in a multi-thread manner.

  As pointed out in the above-described problem, when the contents of the face dictionary are sequentially updated according to the collation result between the face dictionary and the detected face information, the “reproducibility” of the personal recognition result may not be maintained. The reproducibility of personal recognition results largely depends on the face dictionary formation process. It is a clustering problem that the contents of the dictionary change due to a change in the input order of information to be collated and registered.

  For example, it is assumed that A and C are not similar among face information A, B, and C, and B is somewhat similar to A and C to some extent. If the personal recognition processing is executed in the order of face information A, B, and C, B is checked against the dictionary while A is registered in the dictionary, and face information A and B are determined to be similar. Conversely, if the personal recognition processing is executed in the order of face information C, B, A, B is checked against the dictionary with C registered in the dictionary, and it is determined that face information B and C are similar. . Thus, the face information B can be determined as the same person in both A and C depending on the existing dictionary information. When face detection is performed in a multi-thread manner, the detection order of face information is different each time. Therefore, in the configuration as in Patent Document 1, the face recognition result of each face information is different as the detection order is changed. This embodiment solves the problem and improves the “reproducibility” of the personal recognition result.

  FIG. 1 is a block diagram illustrating a hardware configuration of the image processing apparatus. Examples of the image processing apparatus include a PC and a smartphone. In the present embodiment, the image processing apparatus is a PC.

  A CPU (central processing unit / processor) 101 controls the image processing apparatus 100 as a whole, and implements the operation of the present embodiment by reading a program stored in the ROM 102 into the RAM 103 and executing it, for example. In FIG. 1, there is one CPU, but it may be composed of a plurality of CPUs. The ROM 102 is a general-purpose ROM, and stores programs executed by the CPU 101, for example. The RAM 103 is a general-purpose RAM, and is used as a working memory for temporarily storing various types of information when the CPU 101 executes a program, for example. The HDD (hard disk) 104 is a storage medium (storage unit) for storing a database that holds processing results such as image files, image data, and image analysis, a template used by an album creation application, and the like.

  The display 105 displays the user interface (UI) of the present embodiment and the layout result of the image data to the user. The keyboard 106 and the pointing device 107 receive an instruction operation from the user. The display 105 may have a touch sensor function. The keyboard 106 is used when, for example, the user receives an input from the user such as the number of spreads of an album to be created on the UI displayed on the display 105. The pointing device 107 is used when the user clicks a button on the UI displayed on the display 105, for example.

  The data communication unit 108 communicates with an external device via a wired or wireless network. For example, the data communication unit 108 transmits data laid out by the automatic layout function to a printer or server that can communicate with the image processing apparatus 100. The data bus 109 connects the blocks in FIG. 1 so that they can communicate with each other.

  The album creating application according to the present embodiment is stored in the HDD 104, and the user double-clicks the application icon displayed on the display 105 and starts up as will be described later.

  FIG. 2 is a block diagram showing the software configuration of the album creation application. FIG. 2 particularly shows a software block diagram relating to an automatic layout processing unit for executing an automatic layout function. The album creation condition designating unit 201 outputs album creation conditions according to a user's UI operation described later by the pointing device 107 to the automatic layout processing unit surrounded by a thick solid line frame in FIG.

  The image acquisition unit 202 acquires the image data group designated from the album creation condition designation unit 201 from the HDD 104. The image data group may be acquired from a server or SNS on the network via the data communication unit 108. The acquired image data groups are sorted in order of shooting date and time, and serial numbers are associated. By using the serial number, it becomes easy to specify information when processing the designated image data. The image conversion unit 203 converts image data used for subsequent processing into a desired number of pixels and color information. In this embodiment, the image data is converted into analysis image data of 420 pixels of the short side and sRGB color information.

  The image analysis unit 204 executes acquisition of a feature amount described later from the analysis image data. The person information analysis unit 205 executes face detection and individual recognition processes. The image analysis unit 204 also acquires data associated with the image data acquired from the HDD 104, for example, acquisition of the shooting date and time from Exif information. The image classification unit 206 executes scene division and scene classification described later using the shooting date / time information, the number of shots, detected face information, and the like for the image data group. A scene is a shooting scene such as a trip, daily life, wedding or the like. The image scoring unit 208 performs scoring so that image data suitable for the layout has a high score.

  The main character information input unit 207 inputs the main character ID (identification information) instructed from the album creation condition specifying unit 201 to the image scoring unit 208. The image score unit 208 is configured to add a higher score to the image data including the main character ID input from the main character information input unit 207.

  The spread assignment unit 210 divides (groups) the image data group and assigns it to each spread of the album. The spread number input unit 209 inputs the number of spreads of the album designated by the album creation condition designation unit 201 to the spread allocation unit 210. The spread assignment unit 210 divides the image data group according to the input number of spreads, and assigns a part of the image data group (also referred to as division or image group) to each spread. The image selection unit 211 adds image data corresponding to the number of slots designated by the album creation condition designation unit 201 from a part of the image data group assigned to each spread by the spread assignment unit 210 by the image scoring unit 208. Select based on score obtained.

  The image layout unit 213 determines the layout of the image data. The template input unit 212 inputs a plurality of templates corresponding to the template information specified from the album creation condition specifying unit 201 to the image layout unit 213. The image layout unit 213 selects a template suitable for the image data selected by the image selection unit 211 from a plurality of templates input from the template input unit 212, and determines the layout of the image data. The layout information output unit 216 outputs layout information to be displayed on the display 105 in accordance with the image data layout determined by the image layout unit 213. The layout information is, for example, bitmap data obtained by laying out the image data selected by the image selection unit 211 on the selected template.

  The image correction unit 215 performs dodging correction, red-eye correction, and contrast correction. The image correction condition input unit 214 inputs the image correction ON / OFF condition designated by the album creation condition designation unit 201 to the image correction unit 215. The image correction unit 215 performs correction on the image data when the image correction condition is ON, and does not execute correction when the image correction condition is OFF. Note that the image correction unit 215 performs correction on the image data input from the image conversion unit 203 in accordance with ON / OFF of the correction. The number of pixels of the image data input from the image conversion unit 203 to the image correction unit 215 can be changed according to the layout size determined by the image layout unit 213.

  When the album creation application is installed in the image processing apparatus 100, a startup icon is displayed on a top screen (desktop) of an OS (operating system) that runs on the image processing apparatus 100. When the user double-clicks the start icon on the display 105 displayed on the display 105 with the pointing device 107, the album creation application program stored in the HDD 104 is loaded into the ROM 102. When the program in the ROM 102 is read by the CPU 101 into the RAM 103 and executed, the album creation application is activated.

  FIG. 3 is a diagram illustrating an example of a display screen 301 provided by the album creation application.

  A display screen 301 is displayed on the display 105. The user sets album creation conditions, which will be described later, via the display screen 301, and the album creation condition designating unit 201 acquires the contents set by the user.

  A path box 302 on the display screen 301 displays a storage location (path) in the HDD 104 of the image data group to be created. A folder selection button 303 displays a folder including an image data group to be created as an album creation target in a tree structure so that the user can select it by clicking on the pointing device 107 from the user. A folder path including the image data group selected by the user is displayed in the path box 302.

  The hero designation icon 304 is an icon for the user to designate a hero, and a person's face image is displayed as an icon. A plurality of different face image icons are displayed side by side on the main character designation icon 304, and can be selected by the user by clicking with the pointing device 107. Each face image is registered in association with a personal ID. The hero designation icon 304 is used to specify the hero who is the central person from the person shown in the image represented by the image data to be analyzed. For example, the face image is detected in advance from the image data and registered in the face database in association with the personal ID. The hero designation icon 304 can be automatically set, for example, as a face image of a person selected by a user among face images registered in the face database.

  The spread number box 305 accepts the setting of the number of spreads of the album from the user. The user inputs a number directly into the spread number box 305 via the keyboard 106 or inputs a number from the list into the spread number box 305 using the pointing device 107.

  The template designation icon 306 displays an illustration image for each template taste (pop style, chic style, etc.). A plurality of template icons are displayed side by side on the template designation icon 306 and can be selected by the user by clicking with the pointing device 107. A check box 307 receives an ON / OFF designation for image correction from the user. A state with a check is a state in which image correction ON is designated, and a state without a check is a state in which image correction OFF is designated. The UI configuration screen 301 is configured to be able to set ON / OFF for a plurality of correction processes collectively, but may be configured to be able to set ON / OFF for each individual correction process.

  When the user presses an OK button 308, the album creation condition designating unit 201 acquires the contents set on the display screen 301. The album creation condition designation unit 201 outputs the acquired setting contents to the automatic layout processing unit of the album creation application. At this time, the path input in the path box 302 is transmitted to the image acquisition unit 202. The personal ID of the main character selected by the main character specifying icon 304 is transmitted to the main character information input unit 207. The spread number input in the spread number box 305 is transmitted to the spread number input unit 209. The template information selected by the template designation icon 306 is transmitted to the template input unit 212. The image correction ON / OFF designation in the image correction check box 307 is transmitted to the image correction condition input unit 214. A reset button 309 on the display screen 301 is a button for resetting each setting information on the display screen 301.

  FIG. 4 is a flowchart showing the processing of the automatic layout processing unit of the album creation application. Note that the flowchart of the present application is realized, for example, when the CPU 101 reads a program stored in the HDD 104 into the ROM 102 or the RAM 103 and executes the program.

  Here, the automatic layout process will be described with reference to FIG.

  In step S401, the image conversion unit 203 generates analysis image data. The image conversion unit 203 converts each image data of the image data group of the HDD 104 designated by the album creation condition designating unit 201 into analysis image data having a desired number of pixels and color information. In this embodiment, the analysis image data is converted to 420 pixels of short sides and sRGB color information.

  In step S402, the image analysis unit 204 analyzes and acquires the image feature amount. The image analysis unit 204 acquires the shooting date and time from, for example, Exif information attached to each image data acquired from the HDD 104. Further, the image analysis unit 204 acquires a feature amount from the analysis image data generated in S401. For example, the feature amount is in focus. A Sobel filter is generally known as an edge detection method. Edge detection is performed by a Sobel filter, and the edge gradient (luminance gradient) is calculated by dividing the luminance difference between the start point and the end point of the edge by the distance between the start point and the end point. From the result of calculating the average inclination of the edges in the image, an image with a large average inclination can be considered to be in focus than an image with a small average inclination. If a plurality of threshold values having different values with respect to the inclination are set, it is possible to output an evaluation value of the focus amount by determining which threshold value or more. In the present embodiment, two different threshold values are set in advance, and the focus amount is determined in three stages of ◯ Δ ×. For example, the focus inclination to be adopted for the album is determined as ◯, the allowable focus inclination is determined as Δ, the unacceptable inclination is determined as X, and each threshold is set in advance. The threshold setting may be provided by, for example, the creator of the album creation application, or may be set on the user interface.

  In step S403, the person information analysis unit 205 performs face detection on the analysis image data generated in step S401. Here, a known method can be used for the face detection process, for example, Adaboost that creates a strong classifier from a plurality of weak classifiers prepared. In this embodiment, a face image of a person (object) is detected by a strong classifier created by Adaboost.

  The person information analysis unit 205 extracts a face image and acquires the upper left coordinate value and the lower right coordinate value of the detected face image position. By having these two types of coordinates, the image analysis unit 204 can acquire the position of the face image and the size of the face image. Here, not only the face but also animals such as dogs and cats, flowers, food, buildings, figurines and other objects such as objects to be detected instead of faces may be used to create a strong discriminator using Adaboost. Thereby, the image analysis unit 204 can detect an object other than the face image.

  In S404, the person information analysis unit 205 performs personal recognition by comparing the similarity between the face image detected in S403 and the representative face image stored for each personal ID in the face dictionary database. The person information analysis unit 205 sets the ID having the highest similarity that is equal to or higher than the threshold as the ID of the detected face image. When the highest similarity is less than the threshold, the person information analysis unit 205 registers the extracted face image as a new face in the face dictionary database as a new personal ID. Here, not only the face but also animals such as dogs and cats, flowers, food, buildings, figurines, and the like may be used as recognition targets instead of faces. For example, when the person information analysis unit 205 sets a dog as a detection target, it is possible to compare similarities so as to identify dog breeds.

  A detailed configuration of the person information analysis unit 205, which is one of the characteristic configurations of this embodiment, will be described. The person information analysis unit 205 executes the face detection process in S403 of FIG. 4 and the personal recognition process in S404.

  FIG. 11 is a block diagram showing a detailed configuration of the person information analysis unit 205. In the present embodiment, it is assumed that 1401 to 1403 in FIG. 11 are implemented by multithreads, and 1404 to 1406 are implemented by single threads. Further, in the present application, even if the detection order of face information is different each time due to multi-threaded face detection, by executing 1403 to 1405, “reproducibility” of the personal recognition result is obtained. Can be improved.

  An image information acquisition unit 1401 acquires information of image data input from a user. The image information acquisition unit 1401 can also acquire the image analysis information acquired by the image analysis unit 204. By using the information acquired by the image analysis unit 204 in the person information analysis unit 205, it is possible to eliminate the duplication of processing performed in both of the partial color information acquisition and to increase the speed.

  The face information acquisition unit 1402 detects a face appearing in an image from the information of the image data acquired by the image information acquisition unit 1401 and acquires the detection result. The detection result includes the number of faces, the position information of each face, the size information of each face, the rotation angle of each face, the orientation of each face, the positional relationship between each face, the detection reliability of each face, and the like. The detection reliability is obtained by quantifying the degree of “face-likeness” with respect to an area determined to be a face by the face detection result. Further, the pixel value of the detected face area can also be acquired as a detection result. In the present embodiment, the pixel value of the detected face area is acquired after normalizing information for the personal recognition process described later.

  The face information numbering unit 1403 assigns serial numbers to a plurality of pieces of face information acquired by the face information acquisition unit 1402. In this embodiment, an image serial number that is information between image data and a face serial number that is information in the image data are managed in association with each acquired face information. The serial number associated with the image data is used as the image serial number. The face serial numbers are sorted according to the detection reliability acquired at the time of face detection in S403, and numbered for each sorted face.

  For example, the number of faces in the image (the number of faces included in the image data) is three (face A, face B, face C), and the detection reliability of each face is face A = 80, face B = 70, face C = If it is 90, 1 is assigned to face A, 2 to face B, and 0 to face C as the face serial number. It should be noted that the higher the value of the face detection reliability, the more likely it is to be a face. By performing the sorting based on the detection reliability of the face, it is possible to improve the dictionary accuracy of the personal recognition process in S404 described later. In addition, a threshold can be set for the detection reliability after sorting. For example, when the face detection threshold is 75, the number of detected faces after the face detection in S403 is 3, but the face B = 70 is not regarded as a face by the threshold processing, and the total number of faces is 2. Further, from the face orientation information obtained at the time of face detection in S403, the detected face may not be numbered depending on the face orientation. For example, when face A: front, face B: front, face C: profile, and only the face facing front is the numbering target, the number of detected faces after the detection execution was three, but the face orientation was used. The total number of faces is 2 by the filtering process.

  The information management unit 1404 manages the plurality of face information acquired by the face information acquisition unit 1402 and the image serial number and face serial number acquired by the face information numbering unit 1403 in association with each other. The information management unit 1404 sequentially stores face related information (combination information of face information and serial number) in the ROM 102 and RAM 103. If the storable areas of the ROM 102 and RAM 103 are reduced due to the face-related information stored, or if there are other storage target data in the ROM 102 and RAM 103, they can be stored in the HDD 104. In the present embodiment, as the storage information, not only can each face information be individually stored, but also a data table for managing the stored face group can be stored.

  FIG. 12A shows an example of a data table.

  As shown in FIG. 12A, when the processing of S1205 and S1206 in FIG. 12 to be described later is completed, the image serial number and the face serial number are managed in association with each other. The data table is stored in the order of the face related information acquired by the information management unit 1404, but the image serial number may not be stored in ascending order depending on the system configuration and the performance of the CPU 101. In this case, the missing number may be stored in the data table, or the missing number may be skipped. In this embodiment, when face information is not detected from image data, a missing number is written (the image serial number is written and the face serial number is stored as “−”). In this embodiment, a plurality of faces are detected from the image data, but missing face numbers are skipped for face information removed after the face detection reliability sort is performed due to factors such as the face direction being not front. I do. Information stored in the data table can uniquely identify each face information.

  The order control unit 1405 acquires a plurality of pieces of face related information managed by the information management unit 1404, and performs sort processing using the serial numbers of the face related information. This sort process refers to both a sort process using an image serial number and a sort process using a face serial number. When face-related information is stored in the ROM 102 and RAM 103, necessary information is read from the ROM 102 and RAM 103 and sorted. Similarly, when stored in the HDD 104, necessary information is read from the HDD 104 and sorted. The necessary information may be all information in the face related information or only a data table (FIG. 12A) for managing the face group.

  In this embodiment, the image serial number is sorted in ascending order, and the face serial number is sorted in ascending order within the same image serial number. As a result of these two sorts in ascending order, FIG. 12B is obtained. It should be noted that missing numbers and skipped portions stored in the data table shown in FIG. 12A are absorbed by the order control unit 1405, and only for data to which image serial numbers and face serial numbers are assigned, "Recognition execution order" is assigned.

  The personal recognition information acquisition unit 1406 executes personal recognition processing in the order of the face related information sorted by the order control unit 1405. The personal recognition processing is roughly classified into “similarity calculation processing” and “dictionary registration processing”. The personal recognition information acquisition unit 1406 first creates a new face dictionary. A part or all of the registered face information is recorded in the face dictionary. In other words, even if the detection order of face information changes each time due to multi-threaded face detection, the personal recognition processing is executed in the order of the sorted face-related information. The “reproducibility” of the results can be realized.

  The personal recognition information acquisition unit 1406 first registers the face information input to the first person. The face information of the personal recognition execution order “0” in FIG. At the time of registration, the personal ID is linked to the face related information. In this embodiment, if there is no registered face in the dictionary, personal ID = 0 is assigned to newly registered face information. In this embodiment, a new dictionary is created and the first face information is registered, but it does not matter whether the dictionary is in the initial state. For example, face information may already be registered in the dictionary.

  Next, the personal recognition information acquisition unit 1406 collates the face information input by the second person with the face dictionary.

  Here, regarding the face information of the personal recognition execution order “1” in FIG. 12B, the similarity with the face dictionary including the face information of the personal ID = 0 is calculated. Then, regarding the face information of the personal recognition execution order “1”, the similarity with personal ID = 0 is determined, and it is determined whether the personal ID = 0 is regarded as the same person or a new personal ID = 1 should be allocated. After the determination, the face information of the personal recognition execution order “1” is also registered in the dictionary in association with the personal ID. Thus, the face dictionary is updated with face information acquired from image data input by the user. In this embodiment, registration processing is performed for all face images, but it is also possible to limit the face information to be registered by determining representative face information for a certain personal ID.

  The details of the person information analysis unit 205 have been described above, but by using this configuration to perform face detection in S403 and personal recognition processing in S404, the order in which the personal recognition processing is performed and the registration order of the face dictionary are changed. Increases reproducibility. When face detection processing is executed in multiple threads, the order of face-related information input to the information management unit 1404 may be different for each face detection processing even when the same image data group is handled. There is. However, when the order control unit 1405 executes the sort processing, the individual recognition execution order determined at any time can be given to each face related information. Therefore, it is possible to unify the order of the face related information input to the information management unit 1404.

  FIG. 13 is a flowchart showing face detection processing executed by the person information analysis unit 205.

  In the present embodiment, multithreading is executed during the face detection process. The process shown in FIG. 13 is a flowchart of the face detection process executed by one thread. In this embodiment, the image data group input by the user for album creation is divided into each thread (each core that executes each thread) and is divided into equal parts. For example, when the user specifies that 120 pieces of image data are used for album creation and three cores can be used for face detection at the time of processing, one core uses 40 pieces of image data. deal with. In this case, the number of image data (N) allocated to one thread is N = 40. The designation of the number of cores can be adaptively changed according to an instruction from the OS. The distribution of the image data does not need to be equally divided, and can be adaptively changed according to the progress of each thread.

  In this process, a parameter n indicating the number of processed images described later is initialized to “0”. Furthermore, the parameter m indicating the number of detected faces of each image data is also initialized to “0”.

  In S1501, the parameter n indicating the number of processed images is incremented by “+1”. In step S1502, the image information acquisition unit 1401 acquires information about one piece of image data from the image group assigned to the core. At this time, the image information acquisition unit 1401 can also acquire the image analysis information acquired by the image analysis unit 204. In step S1503, the face information acquisition unit 1402 detects a face area in the image data from the image data information and the image analysis information acquired by the image information acquisition unit 1401. In the present embodiment, the sort processing is executed on the acquired face information with the face detection reliability.

  In step S1504, the parameter m indicating the number of detected faces in the number of detected faces (M) acquired in step S1503 is incremented by “+1”. When a plurality of faces are detected from one image data, the processes of S1505 and S1506 are executed in order from the face information with the highest detection reliability.

  In step S1505, the face information numbering unit 1403 associates an image serial number with the detected face information. In this embodiment, a serial number assigned to each image data in the order of shooting date and time of the image data is associated with the face information as an image serial number. At that time, the value of the parameter n in S1501 and the image serial number associated with the face information in S1505 do not necessarily match. For example, when the image serial numbers processed by a certain core are 4, 8, and 9, parameters may be sequentially assigned from 1 as the parameter n in S1501. In the present embodiment, a value newly assigned to each core (that is, a value that does not match the image serial number) is applied to the parameter n in S1501.

  In step S1506, the face information numbering unit 1403 associates a face serial number with the detected face information. In the present embodiment, since the sort processing is executed on the plurality of face information acquired in S1503, the face serial numbers are linked in ascending order from the face information having a high face reliability.

  In S1507, the information management unit 1404 stores face-related information including the face information acquired in S1503, the image serial number of each face information acquired in S1505, and the face serial number of each face information acquired in S1506. In the present embodiment, the face related information is stored in the RAM 103. When the storable area of the RAM 103 decreases, the storage destination is moved to the HDD 104 or the web server and stored. In addition, a data table serving as a correspondence table of the stored face related information is also stored in the RAM 103.

  The contents of the data table (for example, FIG. 12A) are updated during the processing of S1504 to S1507. In the present embodiment, FIG. 12A before sorting is described as the update target, but the data table after sorting may be updated.

  In S1507a, it is checked whether or not the processing of the number of detected faces (M) is completed, that is, whether or not m = M. If m ≠ M, the process returns to S1504, and if m = M, the process is performed. Advances to S1507b. Although not shown, if a face is not detected from the image data, that is, if M = 0, the processing of S1504 to S1507 is not executed, and the processing moves to the next image data. For example, when two pieces of face information are detected from the image data of the image serial number “0”, the processes of S1504 to S1507 are executed twice. As a result, “image serial number: 0-face serial number 0” and “image serial number: 0-face serial number 1” in FIG. 12A are stored.

  In S1507b, it is checked whether or not the processing relating to the allocated number (N) of image data has been completed, that is, whether or not n = N. If n ≠ N, the process returns to S1501, and n = If N, the process ends.

  In this embodiment, the data table is shared between threads, but a separate data table may be created for each thread. When individual data tables are provided, table integration processing is required. When sharing a data table, it is necessary to avoid an access collision in which data write timings overlap among threads. Similarly, in the table integration process, it is necessary to prevent a mismatch between the results due to a difference between the write timing in each thread and the read timing at the time of integration. As a method for preventing such a collision or a result mismatch, lock processing between threads is executed. For example. If there is data shared between threads, lock the write permission before a thread writes. Other threads wait without writing if the write permission is locked.

  Access collision between threads does not occur due to locking, but deadlock occurs when the locking timings are aligned between threads. In order to prevent deadlock, a control layer for controlling the order of lock execution may be introduced. Even when the lock timings are aligned between threads, the control layer arranges the execution order, so no deadlock occurs. Each core executes the process of FIG. 13, and as a result, the data table of FIG. 12A is generated.

  FIG. 14 is a flowchart showing the personal recognition processing executed by the person information analysis unit 205. In the present embodiment, face detection and personal recognition processing operating in a multi-thread are executed in parallel processing. The personal recognition process itself is not executed by multiple cores. FIG. 14 shows a personal recognition process executed by one thread.

  In step S1508, the order control unit 1405 checks whether there is update information in the data table (for example, FIG. 12A) managed by the information management unit 1404 and the face related information group. Here, as long as there is no update information in the data table and the face related information group managed by the information management unit 1404, the process proceeds to S1509 and waits for the process. In the present embodiment, the standby condition is 10 seconds, and after 10 seconds have elapsed, the standby is terminated and the process returns to S1508. The waiting end condition may be a specified number of seconds, or the waiting may be ended using data update information stored in the RAM 103 or the HDD 104 or on the web server as a trigger. On the other hand, in step S1508, if the order control unit 1405 determines that there is an update of the data table and the face related information group, the process proceeds to step S1510.

  In step S1510, the order control unit 1405 reads a data table (for example, FIG. 12A) stored in the information management unit 1404, and sorts the data table in ascending order based on the image serial number. In this embodiment, the image serial number corresponds to the shooting date and time of the image group input by the user for album creation. By ascending order sorting, each face-related information is arranged in the order of the shooting date and time of the image data in which the face was shown.

  In step S1511, the order control unit 1405 reads the data table (for example, FIG. 12A) stored in the information management unit 1404, and performs ascending sort using the face serial number within the same image serial number. Note that the sort criteria may not be in ascending order.

  In step S <b> 1512, the order control unit 1405 adds personal recognition execution order information to the data table. The added personal recognition execution order is used in S1513 to determine whether to execute the personal recognition in S1514. For example, it is assumed that the state of the data table confirmed to be updated in S1509 is the state shown in FIG. After executing the sort processing of S1510 and S1511, in S1512 the personal recognition execution order is added, and a data table as shown in FIG. 12B can be acquired. Thus, the personal recognition execution order is determined according to the ascending order of the image serial numbers and the sorting result of the face serial numbers.

  In the example shown in FIG. 12A, the face-related information with the image serial number 3 is a missing number, and therefore is not regarded as a target of the personal recognition execution order as shown in FIG.

  Here, for the sake of explanation, it is assumed that the update confirmed in S1509 is in the state of the data table as shown in FIG. In this case, the data table of FIG. 12C is subjected to sort processing by S1510 and S1511. However, in the example shown in FIG. 12C, the face detection result of the image data having the image serial number 6 is not described in the data table. In multi-thread processing, even when image data is evenly distributed to each core, a difference occurs in each processing end time.

  In the example of FIG. 12C, serial numbers 0, 3, 6, 9,... For core A, serial numbers 1, 4, 7, 10,... For core B, serial numbers 2, 5, 8,. It is assumed that the image data 11... Is distributed and the face detection processing flow is executed in each core. The cores A, B, and C do not end face detection processing for image data of serial numbers 0, 1, and 2 at the same time, and the execution end timing depends on the content included in each image data and the state of the terminal at the time of execution. Makes a difference. The execution time in each thread is not reproducible and it is difficult to predict the processing end time. In FIG. 12C, since the core B processes the image data earlier than the cores A and C, it corresponds to the image serial number 7 before obtaining the face detection result of the image data having the image serial numbers 6 and 8. The face information of stored image data is stored.

  The data table shown in FIG. 12D can be acquired by the sorting process and the personal recognition execution order addition process for the data table shown in FIG. In the example shown in FIG. 12D, the personal recognition execution order is not determined for the face related information acquired from the image data having the image serial number 7. In this embodiment, the personal recognition execution order is determined according to the ascending order sorting result of the image serial number and the face serial number. From the state shown in FIG. 12C, the number of faces appearing in the image data having the image serial number 6 is unknown. For this reason, in S1512, the face recognition information acquired from the image data having the image serial number 7 ends with the personal recognition execution order being undetermined until the next table update.

  In step S <b> 1513, the order control unit 1405 determines from the personal recognition execution order whether there is face-related information for which personal recognition information acquisition unit 1406 should perform personal recognition. Specifically, this determination is realized by determining whether or not there is no missing image serial number on the data table. For example, when the data table is in the state shown in FIG. 12B, the image serial numbers 1 to 5 are continuously arranged. Therefore, it is determined that all face-related information corresponding to FIG. 12B can execute personal recognition according to the personal recognition execution order. That is, it is determined that there is face-related information for which personal recognition is to be performed. On the other hand, in the state shown in FIG. 12D, image serial numbers 1-5 are continuously arranged. Therefore, it is determined that the face-related information corresponding to the image serial numbers 1-5 corresponding to FIG. 12D can be personally recognized according to the personal recognition execution order. Thereafter, for example, even if the face detection of the image serial number 8 is finished, the face detection of the image serial number 6 is not processed, and there is a gap in the image serial number. Therefore, until the face detection of the image serial number 6 is processed, the sequence control unit 1405 determines that there is no face-related information for which personal recognition is to be performed (S1513-No). The reproducibility of the execution order of the personal recognition is enhanced by the definition of the order in S1512 and the execution possibility determination in S1513. If it is determined in S1513 that there is no face information that can be used for personal recognition, the process returns to S1509 to wait for information update.

  In step S1514, the personal recognition information acquisition unit 1406 performs personal recognition on executable face-related information. In S1514, the similarity with the face related information that can be executed for the face dictionary is calculated. Here, if the similarity exceeds the threshold, the same personal ID as the registered personal ID that was similar is added to the face related information, and if the similarity is below the threshold, A new unregistered personal ID is added to the face related information. As for the face related information, part or all of the information is stored in the face dictionary, and the face dictionary is updated.

  As described above, in this embodiment, even when face detection processing is performed in a multi-thread manner, the order of face-related information for which personal recognition processing is executed in S1514 is the same as the image data used for face detection. As long as there is, it will always be the same. As a result, the reproducibility of the personal ID associated with the face dictionary and each face related information and finally the album creation result is enhanced. That is, when face detection of each of a plurality of image data is executed multiple times by multi-thread processing, the execution order of the personal recognition processing for the face information detected by the previous detection processing and the face information detected by the second detection processing The order of execution of the personal recognition processing for is the same.

  In the present embodiment, the personal recognition process and the face detection process are executed in parallel. As in this embodiment, there is an advantage that the execution time can be shortened if individual recognition is sequentially performed on face-related information detected from image data that has been executed. In addition, after face detection is performed on all image data, face-related information can be sorted and personal recognition processing can be performed.

  The image analysis unit 204 and the person information analysis unit 205 distinguish the image analysis information acquired in S402 to S404 for each ID for identifying each image data as shown in FIG. For example, as shown in FIG. 5, the shooting date / time information acquired in S402, the focus determination result, the number of face images detected in S403, and the position information are stored in a table format. The face image position information is stored separately for each personal ID acquired in S404.

  In S <b> 405, it is determined whether or not the processing of S <b> 401 to S <b> 404 has been completed for all image data in the image data group of the HDD 104 specified by the album creation condition specifying unit 201. If it is determined that the processing has not ended, the processing from S401 is repeated. If it is determined that the process has been completed, the process proceeds to S406.

  In S406, the image classification unit 206 executes scene division. The scene division refers to managing the image data group designated by the condition designation unit 201 by dividing it into a plurality of image groups. The image classification unit 206 divides the image data group into a plurality of groups based on the time difference between the image data calculated from the shooting date / time information acquired in S402. In the present embodiment, for example, when there is a date when no image is taken between image data, the division is performed based on that portion. The division may be performed according to other criteria. For example, when the division is performed even when the shooting dates are consecutive, if the time difference is more than 16 hours, the division is performed based on that portion. In the case of less than 16 hours, if the time difference from the first shooting to the last shooting on each successive day is less than 4 hours, the image data group for each day is divided as one image group. When the time difference from the first shooting to the last shooting on each day is 4 hours or more and the number of shots on each consecutive day is less than 50, the image data group for each day is regarded as one image group. Divided. On the other hand, if the number is 50 or more, no division is performed. FIG. 6A is a diagram illustrating an example of a result of scene division of an image data group by the above scene division method. In FIG. 6A, the captured image data group is divided into eight image groups. In FIG. 6A, an arrow indicates a group boundary. Note that the thresholds for the difference in imaging date and time and the threshold for the number of captured image data used in the description of the above scene division method are examples, and are not limited to these values.

  In step S407, the image classification unit 206 executes scene classification. Specifically, the image classification unit 206 scores the image group obtained by scene division in S406 for each type of scene, and classifies the determination target group into the scene with the highest score. In the present embodiment, the image group is classified into one of the scene types of travel, daily life, and ceremony. Here, before the automatic layout processing unit of FIG. 4 is started, the user collects and designates a plurality of image data determined to be travel, daily life, and a ceremony scene in advance.

  Here, the designation performed by the user in advance will be described.

  For example, the album creation condition designation unit 201 accepts designation of a plurality of image data that the user has determined to be a travel scene on a user interface screen (not shown). Then, the image analysis unit 204 acquires feature amounts for those image data. Here, the acquired feature amount is, for example, a shooting period, the number of shots, and the number of shots. The shooting period is a time difference from the first shooting to the last shooting of a plurality of image data. The number of shots is the number of image data. The number of photographers is the number of faces photographed in a plurality of image data. As a result, the characteristic amount of the shooting period, the number of shots, and the number of shots is acquired for one image data group made up of a plurality of image data that the user has determined to be a travel scene. Note that only one of these may be used as the feature quantity, or another feature quantity may be used.

  Then, the image analysis unit 204 also performs the acquisition of the feature amount of the shooting period, the number of shots, and the number of shots as described above for other image data groups further designated by the user. Then, the image analysis unit 204 calculates the average value and standard deviation of the shooting period, the average value and standard deviation of the number of shots, and the average value and standard deviation of the number of shots based on the feature amounts acquired from the plurality of image data groups. Ask. FIG. 7 shows the average values and standard deviations corresponding to each of the plurality of types of scenes (travel, daily life, ceremony) obtained as described above, and the image analysis unit 204 stores these values in the ROM 102 or the like. Store in advance in the storage area. Alternatively, it may be incorporated in advance in the album creation application program.

  Each average value and standard deviation shown in FIG. 7 are reflected in selection criteria (image data evaluation criteria) when selecting image data corresponding to each of a plurality of types of scenes. Details thereof will be described later.

Returning to FIG. 4 again, the subsequent processing will be described. After the album creation application is activated, for each group in which the image data group specified by the user in the pass box 302 is divided into scenes in S406, the image classification unit 206 has characteristics of shooting period, number of shots, and number of shots A quantity score is calculated. The image classification unit 206 uses the average value and the standard deviation for each scene shown in FIG. Calculate from That is,
Score = 50− | 10 × (average value−feature) / standard deviation | …… (1)
Average score = (Score for shooting period + Score for number of shots + Score for number of photographers) / Number of features
(2)
It is.

  As a result of the calculation, an average score for each scene of travel, daily life, and ceremony is calculated for the image data for each division. Then, the image classification unit 206 classifies the image data of each group into a scene corresponding to the highest score among the above.

  Each average value and standard deviation corresponding to each of a plurality of types of scenes shown in FIG. 7 is a selection criterion for selecting image data corresponding to each of a plurality of types of scenes. Therefore, the image data can be selected based on a selection criterion appropriate for the type of scene.

  If the scenes are the same, the scenes are classified according to a predetermined scene priority order. For example, in the present embodiment, daily> ceremony> travel priorities are defined, and daily scenes have the highest priority. For example, in the image data group 5 after the scene division of FIG. 6A, the shooting period is 36 hours, the number of shots is 300, and the number of shots is 1.7. The average score of the travel scene calculated by the above formulas (1) and (2) is 45.32, the average score of the daily scene is 18.38, and the average score of the ceremony scene is -29.92. Therefore, the image data group 5 is classified as a travel scene. The image classification unit 206 assigns a scene ID to the classified scene so that the scene can be identified.

  In S408, it is determined whether or not the scene classification in S407 has been completed for all groups obtained by the division in S406. If it is determined that the process has not ended, the process from S407 is repeated. If it is determined that the process has been completed, the process proceeds to S409.

  In step S409, the image scoring unit 208 performs hero setting. The main character setting is executed for the image data group designated by the user, and is performed by one of two types of setting methods, automatic and manual. The image scoring unit 208 determines the number of individual IDs that appear in the image data group, the number of individual IDs that appear in each scene, and the The number of scenes in which the personal ID appears can be acquired. The image scoring unit 208 automatically sets the main character from the information without accepting designation from the user. In the present embodiment, when there are a plurality of scenes, the image scoring unit 208 sets a personal ID that has a large number of appearances in a plurality of scenes as the main character ID, and when there is only a single scene, the number of appearances in a single scene. Personal ID with many characters is set as the main character ID.

  When the user designates the hero designation icon 304, the hero information input unit 207 informs the image scoring unit 208 of the designated personal ID. When the personal ID is specified by the user, the image scoring unit 208 sets the personal ID specified by the user as the main character ID, regardless of the main character ID set automatically. This setting is called manual setting.

  The manual setting can be set by the user on the editing screen once the album creation is completed. In order to simplify the user's trouble on the editing screen, it is possible to devise such as presenting a face image representing a personal ID to the user. At this time, an image serial number or a face serial number may be used to determine a face image representing a personal ID. For example, the face serial number is set from the face detection reliability. If a face image representing a face with high face detection reliability is used, a face image with less blur can be displayed on the editing screen.

  In S410, the image scoring unit 208 performs scoring (evaluation of image data). The scoring is to assign a score evaluated from the viewpoint described later to each image data (scoring), and is referred to when selecting image data to be laid out to be used for layout described later. FIG. 10 is a diagram showing a template group used for image data layout. Each of the plurality of templates included in the template group corresponds to each spread. A template 1001 is a single template, a slot 1002 indicates a main slot, and slots 1003 and 1004 indicate subslots. The main slot 1002 is a main slot (a frame for laying out image data) in the template 1001 and is larger in size than the sub slots 1003 and 1004. The image scoring unit 208 gives each image data both a score for the main slot and a score for the subslot.

  FIG. 8A is a diagram showing the concept of the scoring axis. FIG. 8A shows a travel scene, a daily scene, and a ceremony scene classified by the scene classification of S407. In general, the viewpoints of photographs that are considered preferable for travel scenes, daily scenes, and ceremony scenes are different. The most preferred composition in the travel scene is, for example, a photograph in which a symbolic scenery at the destination and a person who will be the main character of the travel are shown together. In the ceremony scene, for example, a photograph of a composition in which two brides and grooms are close to each other (a photograph in which the distance between the two persons is close) is preferred.

  Also, there is not always one kind of photograph that is considered preferable in each scene. In addition to the above-mentioned portraits and landscape photos, the travel scene is preferably a photo with a face up or side profile, a commemorative photo with a person who accompanied or traveled, a photo of food, buildings, or scenery at the destination. May be considered.

  As described above, preferable photo conditions differ from scene to scene, and there are one or more preferable photo conditions in a scene. In this embodiment, in order to calculate two patterns of scores for the main slot and the subslot for image data, as shown in FIG. 8A, preferable photographic conditions are set in each scene. Set two.

  Before the execution of the automatic layout processing unit of FIG. 4 is started, the user preliminarily stores a plurality of pieces of image data determined to conform to the characteristics of the main slot direction and the sub slot direction of each scene shown in FIG. Collect and specify.

  Here, the designation performed by the user in advance will be described.

  For example, the album creation condition designating unit 201 accepts designation of a plurality of pieces of image data that are determined by the user to be in the main slot direction (or subslot direction) of the travel scene on a user interface screen (not shown). The person information analysis unit 205 acquires the number of faces, the position of the face, and the feature amount of the face size of each designated image data. As a result, for each of a plurality of pieces of image data that the user has determined to be in the main slot direction (or sub slot direction) of the travel scene, for example, the face quantity, face position, and face size feature quantities are acquired. Then, the person information analysis unit 205 obtains the average value and standard deviation of the number of faces, the average value and standard deviation of the face position, and the average value and standard deviation of the face size. The person information analysis unit 205 obtains an average value and a standard deviation as a statistical value of each feature amount as described above for each slot type (main slot and subslot) of each scene. The person information analysis unit 205 stores these values in a storage area such as the ROM 102 in advance. Alternatively, it may be incorporated in advance in the album creation application program.

The image scoring unit 208 can acquire information about which scene each image data to be evaluated belongs to from the result of the scene classification in S407. The image scoring unit 208 uses the average value and the standard deviation obtained in advance corresponding to the scene of the target image data, and the number of faces, the face position, and the face size of the main character ID of the target image data. The average score of the image data to be evaluated is calculated by the equations (3) and (4). That is,
Score = 50− | 10 × (average value−feature) / standard deviation | (3)
Average score = (Face number score + Face position score + Face size score) / Number of feature parameters
...... (4)
It is.

  In the expression (3) which is the evaluation standard of the image data, it is obtained for the scene type corresponding to each image data according to each average value and standard deviation corresponding to each of a plurality of types of scenes shown in FIG. Average value and standard deviation are applied. Therefore, it is possible to select image data according to selection criteria (image data evaluation criteria) appropriate for the type of scene.

  The image scoring unit 208 executes the above scoring for both the main slot and the sub slot. Here, since it is preferable that the image data used for the album is in focus, a predetermined score may be added to the image data of the image ID whose focus feature amount is ○ shown in FIG. good. FIG. 8B shows an example of the result of scoring by the above scoring. For each image ID, the main slot and subslot are scored.

  That is, in this embodiment, conditions suitable for the main slot and subslot are determined for each scene as shown in FIG. 8A, and the user designates image data determined to be suitable for them in advance. doing. The person information analysis unit 205 acquires the feature quantity of the number of faces, the face position, and the face size for the image data designated by the user, and obtains an average value and a standard deviation for each feature quantity. When the automatic layout process of FIG. 4 is started after the album creation application is started, each image data (scene classified) subject to the automatic layout process is set to a user judgment criterion such as the orientation of the main slot. Scoring (similarity) indicating whether they are close is performed. For example, in FIG. 8B, image ID 1 is assigned 20 points for the main slot, and image ID 2 is assigned 45 points for the main slot. This means that, for the main slot, the image ID 2 is closer to the user's criterion.

  Here, a detailed configuration of the image score unit 208 will be described.

  FIG. 15 is a block diagram showing a detailed configuration of the image score unit 208.

  According to FIG. 15, the image group acquisition part 1101 acquires the image data group input from the user. The image group acquisition unit 1101 also acquires the scene division information obtained by dividing the scene in S406 when the image data group is acquired.

  The color analysis information acquisition unit 1102 acquires the image feature amount acquired in S402. The color analysis information acquisition unit 1102 also acquires image data shooting date / time information, and analysis information such as the color and edge of the image data, and the amount of focus. The person-related analysis information acquisition unit 1103 acquires the processing results of face detection in S403 and personal recognition in S404. The person related analysis information acquisition unit 1103 also acquires information on the main character setting in S409.

  The image analysis information storage unit 1104 comprehensively manages various image analysis information acquired by the color analysis information acquisition unit 1102 and the person related analysis information acquisition unit 1103. The image analysis information storage unit 1104 also manages secondary features that can be calculated and estimated from the color analysis information acquisition unit 1102 and the person-related analysis information acquisition unit 1103. For example, the image analysis information storage unit 1104 manages not only the position of each face obtained in S403 but also the distance between each face that can be estimated from the position.

  Note that the image analysis information managed by the image analysis information storage unit 1104 is not limited to the information described above. For example, secondary information other than the information acquired by the color analysis information acquisition unit 1102 and the person-related analysis information acquisition unit 1103 may be managed. The image analysis information storage unit 1104 may use only one of the color analysis information acquisition unit 1102 and the person related analysis information acquisition unit 1103.

  The scene storage unit 1105 stores scene classification methods for determining each type of shooting scene and each shooting scene. In the present embodiment, three types of travel scenes, daily scenes, and ceremony scenes, and a scene classification method for discriminating each scene are stored. For example, the scene storage unit 1105 stores information shown in FIGS. 7 and 8A.

  The scene determination unit 1106 determines the shooting scene of the image data from the image analysis information managed by the image analysis information storage unit 1104. The scene determination unit 1106 acquires a processing result corresponding to the scene classification in S407. In the present embodiment, each image data is classified into a travel scene, a daily scene, and a ceremony scene based on the image analysis information. The shooting scene of each image data is determined by the scene classification process in the scene determination unit 1106.

  The parameter setting unit 1107 sets parameters used for scoring based on the determined shooting scene. In the present embodiment, the parameters are, for example, an average value and a standard deviation for the number of faces / face positions / face sizes in a predetermined scene. As the parameter, a predetermined parameter may be used, or the parameter may be set by the user according to a user's rule of thumb or when the system is executed.

  The scoring units 1108, 1109, and 1110 are given different parameters from the parameter setting unit 1107, respectively. Each scoring unit scores (scores) image data from the parameters set by the parameter setting unit 1107 and the image analysis information acquired by the image analysis information storage unit 1104. In the present embodiment, any of the scores acquired by the scoring units 1108, 1109, and 1110 corresponds to the score for the main slot 1002 and the scores for the subslots 1003 and 1004. In FIG. 15, the number of scoring units is three, but the present invention is not limited to this number. For example, if there are three or more scoring axes required for album creation, scoring is executed by three or more scoring units. If less than three scoring axes are sufficient, scoring is performed by less than three scoring units.

By calculating the score of each image data by the image scoring unit 208 configured as described above, scoring can be performed with a scoring axis corresponding to the shooting scene of the image data. Since scoring axes and parameters are not uniformly determined between shooting scenes, it is possible to acquire image data having an optimal composition and number of people for each shooting scene and slot type.
FIG. 16 is a flowchart showing scoring processing after scene classification executed by the image scoring unit 208.

  In step S1201, the image analysis information storage unit 1104 acquires and sets image analysis information from the color analysis information acquisition unit 1102 and the person related analysis information acquisition unit 1103. In step S <b> 1202, the scene determination unit 1106 sets a shooting scene of image data classified based on the scene classification method prepared in the scene storage unit 1105.

  In S1203, a loop process for acquiring a plurality of scoring axes in the shooting scene is executed based on the shooting scene of the set image data. In the present embodiment, the scoring axis is, for example, scoring for the main slot and subslot. In step S1204, the parameter setting unit 1107 sets parameters required for each scoring axis in the shooting scene of the set image data. In S1205, the scoring units 1108, 1109, and 1110 perform scoring according to the set parameters. In the present embodiment, scoring is calculated by the above-described equations (3) and (4).

  A determination threshold value and determination weight information are associated with the n parameters set in S1204. In the present embodiment, the n parameters are, for example, the number of faces / face position / face size for the main slot described above. As for the determination threshold value and the determination weight information, n pieces corresponding to each parameter in each scoring axis e (i, j) are secured. Here, i corresponds to the shooting scene ID of the image data set in S1202. Further, j corresponds to a plurality of scoring axes in the shooting scene.

  Here, the determination threshold value and the determination weight information will be described.

  The image analysis information acquired in step S1201 is compared with the determination threshold value Th {n, e (i, j)}. Here, if the value of the image analysis information exceeds the determination threshold value Th {n, e (i, j)}, the value W {n, e (i, j)} of the determination weight information is expressed by the equation (4). This is reflected in the corresponding parameter (for example, the score of the number of faces). A value in which the determination weight information is reflected in each parameter of Expression (4) is the final score.

  Thus, not only the score is calculated by the equations (3) and (4), but the value of the determination weight information possessed by each parameter may be reflected in the final score on each score axis. . Based on the determination threshold and the determination weight information as described above, a parameter with a weight of 0 depending on the scoring axis (a parameter that is not used, for example, a face size is not used), or any shooting scene or scoring axis is common. The parameters used in can be set flexibly.

  In this embodiment, a score for the main slot and a score for the subslot as shown in FIG. 8B are given to each image data. By scoring on each scoring axis, it is possible to quantitatively determine how superior or inferior the image data matching the user-desired condition is with other image data. Also, normalization of scores may be performed so that superiority or inferiority can be evaluated even between different scoring axes (for example, between main slots and subslots).

  The remaining processing will be described with reference to FIG. 4 again.

  In S411, the image scoring unit 208 determines whether or not the image scoring of S410 has been completed for all image data in the user-specified image data group. If it is determined that the processing has not ended, the processing from S410 is repeated. If it is determined that the process has been completed, the process proceeds to S412.

  In S <b> 412, the spread allocation unit 210 determines whether the number of scene divisions in S <b> 406 is the same as the number of spreads of the album input from the spread number input unit 209. If it is determined that they are not the same, the process proceeds to S413. If it is determined that they are the same, the process proceeds to S416. For example, as shown in FIG. 6A, when the number of scene divisions is 8, and the number of inputs of the spread number input unit 209 is 8, the process proceeds to S416.

  In S413, the spread allocation unit 210 determines whether the number of scene divisions in S406 is smaller than the number of spreads input from the spread number input unit 209. If it is determined that the number is not small (large), the process proceeds to S415. If it is determined that there are few, the process proceeds to S414. As shown in FIG. 6A, when the number of scene divisions is 8, and the input number of the spread number input unit 209 is 10, the process proceeds to S414.

  In S414, the spread assignment unit 210 executes sub-scene division. The sub-scene division means that the divided scene is further subdivided when the number of scene divisions <the number of album spreads. A case will be described where the number of spreads of the album specified for the number of scene divisions 8 in FIG. FIG. 6 (B) shows the result of dividing the sub-scene of FIG. 6 (A). The number of divisions is set to 10 by dividing at the position indicated by the broken-line arrow.

  The criteria for division will be described. Among the divisions shown in FIG. 6A, a division portion having a large number of image data is searched. Here, in order to increase the number of divisions from 8 to 10, two locations with a large number of image data are determined. In FIG. 6A, image group 5 is the largest, and then image group 1 and image group 2. Image group 1 and image group 2 are the same number, but image group 2 has a larger time difference from the first image to the last image, and therefore image group 2 is set as a division target, and image groups 5 and 2 are divided.

  First, the division of the image group 2 will be described. In the image group 2, there are two peaks of the number of image data, and since these two have different shooting dates, the image group 2 is divided at the location indicated by the broken line arrow in FIG. 6B. Next, the division of the image group 5 will be described. In the image group 5, there are three peaks of the number of image data, which are three consecutive days. There are two places where the shooting date changes, but the division is performed at the broken-line arrows in FIG. 6B so that the difference in the number of sheets after the division is small. As described above, the number of divisions is changed from 8 to 10. In this example, the image is divided at different locations on the shooting date. However, if the location where the number of image data is large is a single day, the location may be divided at the location where the time difference is the largest within the single date.

  In S415, the spread allocation unit 210 executes scene integration. The scene integration means that the divided scenes are integrated when the number of scene divisions> the number of album spreads. FIG. 6C shows the result of scene integration of FIG. The number of divisions is set to 6 by integrating the broken lines.

  Explain the integration criteria. Among the divisions shown in FIG. 6A, a division portion having a small number of image data is searched. Here, in order to reduce the number of divisions from 8 to 6 at two places, two places at which the number of image data is small are determined. In FIG. 6A, the image group 8, the image group 3, and the image group 7 are arranged in ascending order. Image group 3 and image group 7 have the same number of image data. However, since the image group 8 adjacent to the image group 7 is the integration target, the image group 3 is the integration target. As a result, the image group 8 and the image group 3 are integrated.

  First, the integration of the image group 3 will be described. When the time difference between the image groups 2 and 4 before and after the image group 3 is compared, the time difference with the image group 4 is smaller. Therefore, the image group 3 is integrated into the image group 4 as indicated by the broken line in FIG. Next, the integration of the image group 8 will be described. Since there is no subsequent image group, the image group 8 is integrated with the previous image group 7 as indicated by the broken line in FIG.

  In S416, the spread allocation unit 210 executes the spread allocation. By S412 to S415, the number of scene divisions and the number of specified spreads are the same. The spread assignment unit 210 assigns from the top division to the top of the spread in order of shooting date and time.

  In step S417, the image selection unit 211 executes selection of image data. Hereinafter, an example in which four pieces of image data are selected from divided image data groups assigned to a certain spread will be described with reference to FIG.

  The section from start to end shown in FIG. 9A shows the time difference (division shooting period) from the shooting date / time of the first image data to the shooting date / time of the last image data for the group assigned to spread. . A method of selecting the first sheet will be described with reference to FIG. There is one main slot 1002 in the template. Accordingly, as the first image, image data for the main slot is selected from the target group. Of the plurality of pieces of image data corresponding to the divided shooting periods shown in FIG. 9B, the image data with the highest score for the main slot assigned in S410 is selected. For the second and subsequent images, image data for subslots is selected.

  In the present embodiment, image data is selected so as not to concentrate on a part of the shooting period in the image group.

  Hereinafter, a method for subdividing the imaging period for the division in this embodiment will be described. As shown in FIG. 9C, the photographing period is divided into two. Next, as shown in FIG. 9D, the second image data is selected from the solid line shooting period in which the first image is not selected. Of the plurality of pieces of image data corresponding to the solid line shooting period in FIG. 9D, the image data having the highest score for the subslot is selected.

  Next, as shown in FIG. 9E, the imaging period for each division in FIG. 9D is divided into two. Then, as shown in FIG. 9F, among the plurality of pieces of image data corresponding to the solid line shooting period in which the first and second images are not selected, the image data with the highest score for the subslot is obtained. Selected as the third sheet.

  The case where the image data does not exist during the image data selection target shooting period and therefore cannot be selected will be described by taking the fourth selection as an example. FIG. 9G shows an example in which there is no image data in the hatched shooting period when the fourth image is selected from the hatched shooting period in which image data is not yet selected. In this case, as shown in FIG. 9H, the photographing period for each division is divided into two. Next, as shown in FIG. 9I, among the plurality of image data corresponding to the solid line shooting period in which the first to third images are not selected, the highest image data for the subslot is the fourth image data. Selected as.

  As described above, by selecting image data based on the scores obtained by different scoring axes like the main slot and the subslot, it is possible to give variations to the image data selected for the layout. That is, since image data with different judgment criteria can be selected, an album rich in change can be created.

  The subsequent processing will be described with reference to FIG. 4 again. In S418, the image layout unit 213 determines an image layout. Hereinafter, an example in which the template input unit 212 inputs (a) to (p) in FIG. 10 for a certain spread according to the specified template information will be described.

  Here, the number of slots of the input template is specified as 3 as an example. FIG. 10 (q) shows the result of arranging the three selected image data in order of shooting date and time, and also shows the orientation of the image. That is, in the image data 1006, the orientation of the image is vertical, and the shooting date is the oldest. In the image data 1007, the image orientation is horizontal, and the shooting date and time is the second oldest. In the image data 1005, the orientation of the image is vertical, and the shooting date and time is the newest. Here, the image data 1005 is for the main slot, and the image data 1006 and 1007 are for the subslot. In this embodiment, image data with an older shooting date and time is laid out in the upper left of the template, and image data with a newer shooting date and time is laid out in the lower right. In FIG. 10 (q), the image data 1005 for the main slot has the latest shooting date and time, so the templates of FIGS. 10 (i) to 10 (l) are candidates. Further, since the older image data 1006 for the subslot is a vertical image and the newer image data 1007 is a horizontal image, the template shown in FIG. 10 (i) is converted into the selected three image data. It is determined as the most suitable template, and the layout is determined.

  At this time, the number of slots, vertical and horizontal positions, and positions of the slots match the template of FIG. 10 (i), and the size of the main slot is larger than that of FIG. 10 (i). Suppose a template is searched. That is, when there are a plurality of template candidates based on the order of shooting date and time and the vertical and horizontal positions of the selected image data, in the present embodiment, a template is selected from the score difference of each image data. For example, when the score of the image data of the main slot is larger than the average score of the image data of the plurality of subslots by a predetermined threshold, the template shown in FIG. 17 having a larger main slot is selected. On the other hand, if the threshold is not exceeded, the template shown in FIG. 10 (i) is selected. The above describes the case where another template is searched for the main slot. However, when another template is similarly searched for the subslot, it may be compared with the average score of the other slot.

  In S418, which image data is to be laid out in which slot of which template is determined.

  In step S419, the image correction unit 215 performs image correction. When the image correction ON is input from the image correction condition input unit 214, the image correction unit 215 performs image correction. As image correction, for example, dodging correction, red-eye correction, and contrast correction are executed. The image correction unit 215 does not perform image correction when image correction OFF is input from the image correction condition input unit 214. The image correction can be executed, for example, also on image data having a short side of 1200 pixels and a size converted to an sRGB color space.

  In S420, the layout information output unit 216 creates layout information. The image layout unit 213 lays out the image data subjected to the image correction in S419 for each slot of the template determined in S418. At this time, the image layout unit 213 scales and lays out the image data to be laid out according to the slot size information. Then, the layout information output unit 216 generates bitmap data in which image data is laid out in a template slot.

  In S421, it is determined whether or not the processing of S417 to S420 has been completed for all spreads. If it is determined that the processing has not ended, the processing from S417 is repeated. If it is determined that the process has ended, the automatic layout process of FIG. 4 ends.

  Therefore, according to the embodiment described above, even if the face detection processing is executed by multithread processing, the reproducibility of the personal recognition result can be improved as long as the same data group is used. Further, since the image scoring unit and the image layout unit following this processing also use the result of personal recognition, the improvement in the reproducibility of the personal recognition result is reflected in the album layout result. As a result, even if a face dictionary optimal for the user is created, it is possible to speed up the album creation and improve the reproducibility thereof.

<Embodiment 2>
A common feature of the embodiment of the present invention is that it has a configuration capable of executing multi-thread for each page of an album specified by a user. In the first embodiment, the image data group input by the user for album creation is assigned to the core and face detection is performed. At that time, the reference of the number of image data to be assigned to each core is the shooting date and time, and the processing of each core is performed. The image data was distributed so that there was no bias.

  In this embodiment, after an image data group input for album creation by the user is acquired, event division processing and page allocation processing are executed. This embodiment is different from the first embodiment in that the input image data group is divided into a necessary number of pages before image analysis and evaluation value calculation are executed. The image processing apparatus assigns thread processing to each core in units of pages, and each core executes image analysis processing for an image group belonging to the assigned page.

  Note that the software configuration of the album creating application used in this embodiment is the same as that described with reference to FIG. However, this embodiment is characterized in that the main character information input unit 207 affects the processing of the image selection unit 211 instead of the image scoring unit 208.

  FIG. 18 is a flowchart showing processing executed by the automatic layout processing unit of the album creating application. In FIG. 18, the same processing steps as those already described in the first embodiment with reference to FIG. 4 are denoted by the same step reference numerals, and the description thereof is omitted.

  In step S <b> 1801, the image conversion unit 203 acquires Exif information of image data. The acquired Exif information records the shooting date and time of the image data. Next, in the scene division process of S406, the image group is divided for each scene based on the shooting date and time of each image data acquired in S1801. For the divided scenes, the division / integration processing corresponding to the required number of pages is executed by the processing of S412 to S416. The scene dividing method and the processing method for dividing and integrating the pages into the number of pages are as described in the first embodiment.

  Thereafter, S401 to S405 are executed for all image data, and scene classification is executed for all scenes in S407 to S408.

  Thereafter, in S1802, the image scoring unit 208 performs scoring (evaluation of image data). Although this execution method is the same as that described in the first embodiment, the image scoring unit 208 of this embodiment does not set the main character and does not reflect the main character information in scoring.

  When scoring (evaluation of image data) is completed for all image data, S417 to S418 and S421 are executed until full spread is completed, and then the main character is set in S409.

  In step S <b> 1803, the image selection unit 211 replaces image data in consideration of the main character setting information in step S <b> 409 from the layout information acquired in the image layout determination in step S <b> 418.

  In this embodiment, the main character information is not reflected at the time of image selection in S417, and the image data selection process is executed based on the scoring result according to the evaluation criteria such as the composition information of the image data and the person balance. Therefore, there is no guarantee that the main character is reflected in the image data selected in the image layout determination in S418. For this reason, even if image data having a good composition is arranged in the main slot, there is a possibility that the image data is not preferable for the user. In S1803, in response to the result of the main character setting in S409, the image data that is not preferable for the user in the selected image data and the image data that is preferable for the user in the selection candidate image data are switched.

  For example, if the selected image data is “image data with passers-by”, “image data showing the main character” determined to be replaceable based on factors such as image data score, shooting date and composition, etc. The image data can be replaced. From the information that the person is not the main character, it is possible to regard it as a “passerby”.

  The replacement standard may prioritize the main character information and change the next priority according to the slot type. For example, when the image data of the main slot is replaced, the priority of “scoring of image data” may be increased in order to select image data having the same high score as that of the target selected image data. . When replacing the image data in the subslot, increase the priority of the “image data shooting date” in order to select the image data in consideration of the shooting date of the target selected image data and the adjacent selected image data. May be. In any case, an album layout preferable for the user can be completed by a process different from that of the first embodiment by executing the image data replacement process.

  Next, details of processing in units of pages will be described.

  FIG. 19 is a flowchart for executing processing in units of pages in a multi-thread manner.

  In this process, a parameter l indicating the number of processed pages, which will be described later, is initialized to “0”, and a parameter k indicating the number of processed image data of each page is also initialized to “0”.

  In S1901, the parameter l indicating the number of processed pages is incremented by “+1”. In the present embodiment, a page assignment process for assigning image data groups input by the user for album creation to each page is executed in advance. Then, it is assigned to each thread in a processing unit of each page of the album. For example, as shown in FIG. 20, it is assumed that the user creates an album with 9 pages. In this case, the spread allocation unit 210 executes page allocation according to the shooting date / time of the image data as shown in FIG. 20, and in order from page 1 to page 9, 20, 30, 25, 30, 25 Assign 10 sheets, 20 sheets, 30 sheets, and 10 sheets. Here, the deviation in the number of assigned image data for each page is a result of the influence of the density of the photographing date and time intervals of the image group and the grouping of scenes. Each page is assigned to each core according to the instruction of the OS. The designation of the number of cores may be adaptively changed according to an instruction from the OS. Further, the number of pages allocated to each core may be uniform, or there may be a deviation according to the system configuration and the performance of the CPU 101.

  In the example illustrated in FIG. 20, pages of A = {1, 4, 7}, B = {2, 5, 8}, and C = {3, 6, 9} are respectively included in the three cores A, B, and C. Assigned. That is, each core is in charge of processing for three pages. Therefore, if the example shown in FIG. 20 is applied, three loops relating to S1901 to S1907a are executed in each thread. That is, the process allocation page number L is L = 3.

  In S1902, the parameter k indicating the number of processed image data among the number of image data allocated to each page is incremented by “+1”. If the example shown in FIG. 20 is applied, in the case of the processing of the core A, 20 sheets of image data are processed on the first page, 20 sheets on the 4th page, and 30 sheets on the 7th page. Therefore, the number K of assigned image data in the first loop is K = 20, K = 30 in the second loop, and K = 20 in the third loop.

  In step S1903, image analysis processing is executed. The execution method of the image analysis processing is the same as the processing in the image analysis unit 204 described in the first embodiment and the image feature amount acquisition processing in S402. In S1904, face detection processing is executed. The contents of the face detection process are the same as those described with reference to FIG. 13 in the first embodiment.

  In S1904a, it is checked whether or not the processing has been completed for all the image data to be allocated. Here, if K = k, it is determined that the process is completed, and the process proceeds to S1905. If K ≠ k, the process is not completed, and the process returns to step S1902.

  In S1905, a scoring process is performed. The scoring process is performed in the same manner as in the image selection unit 211 and S410 described in the first embodiment, except that the main character information is not reflected in the image evaluation value. In S1906, an image selection process is executed. The execution method of the image selection process is the same as the description of the image selection in the image selection unit 211 and S417 described in the first embodiment. In S1907, layout selection processing is executed. The execution method of the layout selection process is the same as the description of the image layout determination in the image layout unit 213 and S418 described in the first embodiment.

  In S1907a, it is checked whether or not the processing for all the assigned pages L has been completed. Here, if L = 1, it is determined that the process is completed, and the process ends. If L ≠ 1, the process returns to step S1901 because the process is not completed.

  As described above, in this embodiment, multi-threading can be executed in units of pages, and an album layout for each page can be created in each thread. Accordingly, unlike the first embodiment, not only face detection but also scoring processing, image selection, and layout selection are executed in a multi-thread manner, so that the processing can be further speeded up.

  Further, in this embodiment, as in the first embodiment, the personal recognition processing is executed in parallel processing for each core moving in multithread. The personal recognition process is as described with reference to FIG. 14 in the first embodiment. As in the first embodiment, the configuration of face detection in S403 and personal recognition processing in S404 is as shown in FIG. The process in the face information numbering unit 1403, the information management unit 1404, and the order control unit 1405 increases the reproducibility of the personal recognition result.

  Next, an album layout replacement process related to the image selection unit 211, which is executed after the personal recognition process is completed for the face related information of all face detection results, will be described.

  FIG. 21 is a flowchart showing details of the album replacement process.

  In S1908, it is checked whether or not the personal recognition process shown in FIG. If it is determined that the personal recognition process is complete, the process proceeds to S1910. If it is determined that the personal recognition process is not complete, the process proceeds to S1909, and the process waits for a predetermined time. In this embodiment, the predetermined time is set to 10 seconds, and after 10 seconds have elapsed, the standby is finished, and the process returns to S1908 to check whether the personal recognition process is finished. In this way, the process waits until the personal recognition process is executed for all the faces in the image group input by the user.

  The standby end condition may be a specified number of seconds other than 10 seconds, or the standby may be ended using data update information on the ROM 102, HDD 104, or web server as a trigger. It is also possible to share the end determination flag between threads and use the trigger to change the flag contents as the personal recognition ends.

  In S1910, the main character is set. The execution method of the hero setting is the same as the hero information input unit 207 and the hero setting process in S409 described in the first embodiment. In the process shown in FIG. 21, when performing the hero setting in S1910, the end of face detection and personal recognition is already determined for all image data input by the user for album creation. The main character can be determined using. Further, since face detection and personal recognition are executed with the same configuration as that shown in FIG. 11 described in the first embodiment, the reproducibility of the execution order of personal recognition and the album creation result is improved.

  In step S1911, the image selection unit 211 performs image data replacement processing using the layout information determined in step S1907. The main character information acquired in S1910 is used as information used for the replacement determination.

  In this process, if the protagonist does not appear in the selected image data selected in S1906, the image data approximated with at least one element from the image data score, the image data shooting date and time, and the composition of the image data are replaced candidates. Let it be image data. Then, if there is image data in which the main character is reflected in the replacement candidate image data, the replacement processing is executed with the selected image data. Regarding each approximate determination condition that is a replacement determination criterion, the weight may be changed between the main slot and the subslot, or each approximate determination condition may be changed between different scenes.

  For example, it is considered that a page on which a travel scene is posted often gives a high score to landscape image data in which a passerby is mixed. Therefore, it is necessary to set a loose approximation condition for each element in the image data exchange for the travel scene. If the approximate condition of each element becomes loose, even if the score is poor, the image data showing the main character is replaced with priority.

  On the other hand, in the wedding scene, there are many pictures taken mainly by the bride and groom in the image group, so there is a case where the image data exchange process according to the personal recognition result is not necessary. It is not necessary to replace the image data of the bride and groom whose face has been unable to be recognized with other image data. Therefore, the wedding scene may be set with strict approximation conditions for each element in the image data exchange. If the approximate condition of each element becomes strict, even if it is determined that the main character is reflected in the image data, if the score is not high, the replacement is not performed unless the shooting date is close.

  Therefore, according to the embodiment described above, the reproducibility of the personal recognition result can be improved even if the album layout is executed by multi-thread from the image analysis of each page. Since the reflection of the personal recognition result is finally executed by the image data replacement process, the album layout result also has reproducibility. Further, by increasing the number of multi-thread processes, it is possible to realize a further increase in speed compared to the first embodiment.

<Embodiment 3>
In this embodiment, when the album creation application is interrupted in the middle of album creation or the application is terminated after creation, the application is re-executed and the album is created again to increase the reproducibility of the results. explain.

  FIG. 22 is a flowchart showing face detection processing executed by the person information analysis unit 205. In this embodiment, as in the first embodiment, face detection is performed in a multi-thread manner. In FIG. 22, the same processing steps as those described in the first embodiment with reference to FIG. 13 are denoted by the same step reference numerals, and the description thereof is omitted.

  After S1501, in S1501a, in response to the count-up of parameter n executed in S1501, it is checked whether image analysis processing including face detection has been performed on the image data indicated by the parameter. For example, it is assumed that the user inputs 200 pieces of image data for creating an album. If the application is terminated when the user analyzes 120 pieces of image data at the time of starting the application last time, the face related information as the analysis result is stored on the HDD 104 or the web server. Similarly, a data table for managing analysis results for 120 sheets is also stored on the HDD 104 or the web server. Therefore, if such stored data can be reused when the next album is created, the corresponding processing can be shortened. For this reason, in S1501a, the data at the time of the last creation that can be used is searched, and it is checked whether or not the image analysis processing including face detection has been performed on the corresponding image data.

  If it is determined that there is information obtained by such image analysis processing, the processing skips S1502 to S1507a and proceeds to S1507b. On the other hand, if it is determined that there is no information obtained by such image analysis processing, the processing proceeds to S1502, and the processing described in the first embodiment is executed.

  When the processing from S1502 to S1506 is executed, the information management unit 1404 stores face related information, which is a combination of the face detection result and the serial number, in the HDD 104 in S1507 '. In the first embodiment, the face-related information is stored in the RAM 103 as much as possible. However, in this embodiment, the face-related information is stored in the HDD 104 unconditionally. By storing in the HDD 104, past analysis results can be reused when the application is restarted. In S1507 ', the data table in which the face detection result of the image data matching the count value of the parameter n obtained in S1501 can be stored in the HDD 104 in the same manner. Further, when the association between the user and the image analysis result can be realized by the management system, the data table can be stored on the web server.

  As described above, in this embodiment, it is confirmed whether past analysis results remain as compared with the first embodiment. For this reason, the same results can be obtained for image analysis and face detection when the album is recreated, so that the execution of the processing can be omitted for the same image data when recreating. In this way, the processing time is shortened by reusing past analysis results.

  FIG. 23 is a flowchart showing the personal recognition processing executed by the person information analysis unit 205. In this embodiment, face detection (multi-thread) and personal recognition processing are executed in parallel as in the first embodiment. In FIG. 23, the same processing steps as those described in the first embodiment with reference to FIG. 14 are denoted by the same step reference numerals, and the description thereof is omitted.

  In step S1508a, the personal recognition information acquisition unit 1406 performs face dictionary update processing. The face dictionary can also store the previous analysis result in the HDD 104. In this embodiment, the face dictionary is updated to a new state in order to improve the reproducibility of the personal recognition result. A mechanism for rolling back to a stage where the face dictionary and the data table can be matched and matched may be included. In this case, in S1508a, the personal recognition information acquisition unit 1406 performs an update process for returning the face dictionary to a state where consistency with the previous creation time point can be obtained.

  In step S1508b, the order control unit 1405 updates the data table. As shown in FIG. 12D, the personal recognition execution order at the time of the previous creation is stored in the data table. The execution order is not given to the face related information for which the personal recognition execution is completed and the personal ID is determined. In this embodiment, the face dictionary is updated to a new state in S1508a. Accordingly, the personal recognition execution order is also updated to a new state. When the face dictionary is updated to a state where consistency can be obtained by a rollback function or the like, update processing is performed to return the personal recognition execution order to a state where consistency can be obtained with the face dictionary.

  The subsequent processing is as described in the first embodiment.

  Therefore, according to the embodiment described above, compared to the first embodiment, the past analysis result is most utilized, and when the past analysis result remains, each information including the data table is reused when the album is recreated. can do. Thereby, even when the writing of the data table is terminated in the middle due to forced termination of the application or the like, it is possible to cope with an association error with the face related information and improve the reproducibility of the result.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. In this case, the program and the storage medium storing the program constitute the present invention.

101 CPU, 102 ROM, 103 RAM, 104 HDD

Claims (14)

An image processing apparatus that lays out a plurality of image data,
Obtaining means for obtaining a plurality of image data;
Detecting means for detecting information of a specific part of each of the plurality of image data acquired by the acquiring means by multi-thread processing;
Based on the sorting result obtained by sorting the information on the specific part of each of the plurality of image data detected by the detecting means, the specific part of the information on the specific part of the plurality of image data Determining means for determining an execution order of a specific process using the information of the part;
Execution means for executing the specific process according to the execution order determined by the determination means;
An image processing apparatus comprising: a layout unit that lays out the plurality of image data based on a result of the specific processing executed by the execution unit. A dividing unit that divides the plurality of image data acquired by the acquiring unit into a plurality of image groups;
Assigning means for assigning a plurality of image groups divided by the dividing means to a plurality of cores;
2. The image according to claim 1, wherein the detection unit detects information of the specific portion with respect to image data included in the plurality of image groups by the plurality of cores allocated by the allocation unit. Processing equipment. The dividing unit divides the plurality of image data into the plurality of image groups according to a predetermined criterion and allocates the plurality of image data to the plurality of pages when the plurality of image data are laid out over a plurality of pages. ,
The image processing apparatus according to claim 2, wherein the assigning unit assigns the image data assigned to each of the plurality of pages by the dividing unit to the plurality of cores.   The image processing apparatus according to claim 3, wherein the predetermined reference includes shooting dates and times of the plurality of image data.   An exchanging unit for exchanging at least a part of the plurality of image data laid out by the layout unit after the information on the specific part of the image data included in the plurality of image groups is detected by the detecting unit; The image processing apparatus according to claim 3, further comprising: And further comprising storage means for storing information on the specific part detected by the detection means,
When performing a second layout using a plurality of image data acquired by the acquisition unit, the layout unit executes the layout using the information on the specific part stored by the storage unit. The image processing apparatus according to any one of claims 1 to 5.   The detection by the detection means is re-executed based on the information of the specific part stored in the storage means when the detection by the detection means is interrupted and then re-executed. 6. The image processing apparatus according to 6.   The image processing apparatus according to claim 6, wherein the storage unit includes an HDD.   The image processing apparatus according to claim 1, wherein the information on the specific part is information on a person's face included in each image data. First numbering means for assigning a number to the acquired plurality of image data in order to identify the plurality of image data;
Second numbering means for assigning a number to the detected face in order to identify the face detected by the detection means;
The determining means determines the execution order based on the number of image data assigned by the first numbering means and the face number assigned by the second numbering means. The image processing apparatus according to claim 9.   The image processing apparatus according to claim 10, wherein the execution unit recognizes an individual corresponding to the detected face according to the execution order.   When detection processing for detecting information on a specific portion of each of the plurality of image data is performed again by multi-thread processing, the execution order of information on the specific portion of each of the plurality of image data detected in the previous detection processing is performed 12. The image processing apparatus according to claim 1, wherein the execution order of information of specific portions of each of the plurality of image data detected by the second detection process is the same. .   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 12. An image processing method in an image processing apparatus for laying out a plurality of image data,
An acquisition step of acquiring a plurality of image data;
A detection step of detecting information of a specific portion of each of the plurality of acquired image data by multi-thread processing;
The information on the specific part with respect to the information on the specific part of each of the plurality of image data based on the sorting result obtained by sorting the information on the specific part of each of the detected plurality of image data. A determination step for determining the execution order of a specific process using
An execution step of executing the specific process according to the determined execution order;
A layout step of laying out the plurality of image data based on a result of the specific processing executed in the execution step.

Images