JP5387103B2 - Image processing device - Google Patents

Description

  The present invention relates to an image processing apparatus.

  The following image search system is known. This image search system searches for an image based on information on the size of the main subject (for example, Patent Document 1).

JP 2008-103850 A

  However, in the conventional image search system, it is possible to search for an image based on the main subject, but in order to find an image desired by the user from the search results, the user needs to check the images one by one. It took time and effort.

An image processing apparatus according to the present invention includes main subject specifying means for specifying a main subject in an image, and information detection for analyzing the image and detecting a plurality of pieces of information relating to a shape or color for specifying a sensitivity expression of the main subject. Means, a sensitivity expression determination means for determining a sensitivity expression of the main subject based on a plurality of information detected by the information detection means, and a main subject determination means for determining whether or not the main subject is a person. When the main subject determination unit determines that the main subject is a person, the information detection unit stores a plurality of pieces of information regarding the degree of smile of the person, the pose of the person, the background color, and the background brightness. If it is detected as information and the main subject determination means determines that the main subject is not a person, it relates to the background color, background brightness, and image complexity. And detecting the information as a plurality of information.
An image processing apparatus according to the present invention includes main subject specifying means for specifying a main subject in an image, and information detection for analyzing the image and detecting a plurality of pieces of information relating to a shape or color for specifying a sensitivity expression of the main subject. Means, a sensitivity expression determination means for determining a sensitivity expression of the main subject based on a plurality of information detected by the information detection means, and a main subject determination means for determining whether or not the main subject is a person. When the main subject determining unit determines that the main subject is a person, the information detecting unit detects information about each of the person's smile level and the person's pose as a plurality of pieces of information, and the main subject determining unit However, if it is determined that the main subject is not a person, information on each of the background color and the background brightness should be detected as a plurality of pieces of information. And butterflies.

  According to the present invention, the user can easily find an image by using the determination result of the sensitivity expression of the main subject for the search of the image.

It is a block diagram which shows the structure of one Embodiment of an image processing apparatus. It is a figure which shows the specific example of a search condition designation | designated screen. It is a 1st figure which shows the example of a list display of a search result. It is a flowchart figure which shows the flow of a sensitivity expression determination process. It is a flowchart figure which shows the flow of an image file search process. It is a 2nd figure which shows the example of a list display of a search result.

  FIG. 1 is a block diagram showing a configuration of an embodiment of an image processing apparatus according to the present embodiment. As the image processing apparatus 100, for example, a personal computer (personal computer) is used. The image processing apparatus 100 includes an operation member 101, a connection IF (interface) 102, a control device 103, an HDD (hard disk drive) 104, and a monitor 105.

  The operation member 101 includes various devices operated by the user, such as a keyboard and a mouse. The connection IF 102 is an interface for connecting an external device such as a digital camera. For example, a USB interface for performing a wired connection with a digital camera or a video camera, a wireless LAN module for performing a wireless connection, or the like is used. The In the present embodiment, for example, image file data is captured from the digital camera via the connection IF 102.

  The control device 103 includes a CPU, a memory, and other peripheral circuits, and controls the entire image processing device 100. The memory constituting the control device 103 is a volatile memory such as SDRAM, for example. This memory is used as a work memory for the CPU to expand the program when the program is executed, or as a buffer memory for temporarily recording data.

  The HDD 104 is a recording device for recording an image file taken in via the connection IF 102, various programs executed by the control device 103, and the like. For example, the HDD 104 records program data for executing processing to be described later. This program is provided by being recorded on a storage medium such as a CD-ROM or a DVD-ROM. In the image processing apparatus 100, the user installs program data in the HDD 104 using a storage medium, so that the control apparatus 103 can execute the program.

  The monitor 105 is a liquid crystal monitor, for example, and displays an image of display data output from the control device 103.

  In the present embodiment, the user can instruct each image file recorded in the HDD 104 to record information relating to the emotional expression of the main subject in the captured image in association with the image file. The emotional expression of the main subject is an expression of the emotion of the main subject that can be grasped from the photographed image such as “fun”, “serious”, “calm” and the like.

  When the above instruction is given by the user, the control device 103 identifies the main subject in the captured image of each image file recorded in the HDD 104. In the present embodiment, the control device 103 determines that the person is the main subject when the captured image is captured in the portrait mode, and performs a known face recognition process on the captured image. By executing, the person's face is specified as the main subject. On the other hand, when the shot image is shot in a shooting mode other than portrait mode, it is determined that the main subject is other than a person, and it is determined that the main subject exists in the AF area in the shot image. Then, the subject included in the AF area is specified as the main subject.

  Information regarding the shooting mode at the time of shooting is recorded as header information of the image file on the digital camera side, and the control device 103 refers to the header information to determine whether the shot image was shot in the portrait mode. Judging.

  If the main subject in the captured image is identified, the control device 103 analyzes the image based on the identification result, performs a sensitivity expression determination process for the main subject, and writes the determination result in the header information of the image file. As a result, the information relating to the sensitivity expression of the main subject is associated with the image file. In the present embodiment, a method for determining whether the sensitivity expression of a main subject is “fun” or “serious” and associating the determination result with an image file will be described.

  In the present embodiment, the control device 103 detects a plurality of information that is a factor for determining the sensitivity expression of the main subject, and for each factor, the score becomes higher as the sensitivity expression of the main subject is “fun”. For example, scoring is performed with a maximum of 10 points so that the more serious the score is. Then, the control device 103 determines the sensitivity expression of the main subject based on the total score. A plurality of pieces of information that are factors for performing the scoring differ depending on the result of specifying the main subject described above. Hereinafter, a method for determining the sensitivity expression of the main subject according to the result of specifying the main subject will be described.

  First, a determination method when a human face is specified as a main subject will be described. In this case, the control device 103 determines the sensitivity expression of the main subject by scoring as follows.

  The control device 103 performs a known smile recognition process to detect the smile level of the face of the person identified as the main subject, and performs scoring by evaluating the detected smile level in 10 stages. For example, 10 points are added when the smile level is the highest, and 0 points are added when the smile level is the lowest.

  In addition, the control device 103 recognizes the pose of the person by performing a known shape recognition process on the peripheral image of the face of the person who is the main subject, and performs scoring according to the recognized pose. For example, 1 point is added when a person is a piece (V sign), 5 points is added when a person is shaking hands, and 10 points is added when a person is aged.

  The control device 103 detects the background color and brightness of the image, and performs scoring based on the detection result image. For example, if the background color is red, 8 points are added, and if the background color is blue, 3 points are added. If the background brightness is equal to or higher than a predetermined value, 8 points are added. If the background brightness is lower than the predetermined value, 3 points are added.

  When the total score obtained as a result of scoring in this way is equal to or greater than a predetermined threshold value, the control device 103 determines that the emotional expression of the main subject of the image is “fun”. On the other hand, when the total score is less than the predetermined threshold, it is determined that the sensitivity expression of the main subject of the image is “serious”.

  Next, a determination method when a subject in an AF area other than a human face is specified as a main subject will be described. In this case, the control device 103 determines the sensitivity expression of the main subject by scoring as follows.

  The control device 103 detects the color and brightness of the image in the AF area including the main subject, and performs scoring based on the detection result image. For example, if the color of the image in the AF area is red, 8 points are added, and if the color is blue, 3 points are added. If the brightness of the image in the AF area is greater than or equal to a predetermined value, 8 points are added, and if the brightness is less than the predetermined value, 3 points are added.

  In addition, the control device 103 detects the complexity of a known image, and performs scoring based on the complexity. For example, when the image is complicated, such as when the image includes a floral pattern, 10 points are added, and when the image is simple, such as when the image is monotone, 0 points are added.

  When the total score obtained as a result of scoring in this way is equal to or greater than a predetermined threshold, the control device 103 determines that the emotional expression of the main subject of the image is “fun”. On the other hand, when the total score is less than the predetermined threshold, it is determined that the sensitivity expression of the main subject of the image is “serious”.

  Note that the scoring method described above is an example, and in accordance with the rule that scoring is performed so that the score is higher as the emotional expression of the main subject is “fun” and the score is lower as “serious”. The score to be added in each case may be different from the above score. In addition to the above factors, the scoring may be performed in consideration of other factors that can be used to determine the sensitivity expression of the main subject. For example, event information when an image is taken is detected, and if the event is a “fun” event such as an athletic meet, a high score is added, and if it is a “serious” event such as a graduation ceremony What is necessary is just to add a low point.

  The control device 103 associates the sensitivity expression of the main subject with the image file by recording the information related to the sensitivity expression of the main subject determined by the above-described determination processing of the sensitivity expression of the main subject in the header information of the image file. In this embodiment, the user can search for an image file based on information related to the sensitivity expression of the main subject recorded in the header information of the image file. The image file search process will be described below.

  When searching for an image file in order to view a desired image file from among the image files recorded in the HDD 104, the user searches for the emotional expression of the main subject in the captured image (search condition of the image). ) To search for image files.

  For example, the user designates a search condition on a search condition designation screen as shown in FIG. In the example shown in FIG. 2, “fun” and “serious” can be specified as search conditions among the sensitivity expressions of various main subjects on the search condition specification screen. The user can specify a search condition by checking a check box corresponding to a desired one of these emotional expressions. The user can instruct to start searching for an image file that matches the specified condition by pressing the “Search” button 2a on the search condition specifying screen.

  When the control device 103 detects that the “search” button 2 a has been pressed by the user, the control device 103 reads out information related to the emotional expression of the main subject from the header information of the image file recorded in the HDD 104, and Only those image files whose emotional expressions match the emotional expressions specified on the search condition designation screen are extracted. Then, the control device 103 displays a list of images of the extracted image file on the monitor 106. For example, as shown in FIG. 2, when “fun” is specified as a search condition by the user, the control device 103 records the header information from the image files recorded on the HDD 104. An image file whose information related to the sensitivity expression of the main subject is “fun” is extracted, and the extracted images are displayed in a list as shown in FIG.

  Through the above processing, the user can search the image file recorded in the HDD 104 based on the emotional expression of the main subject. For example, an image file in which the emotional expression of the main subject is “fun” or “serious” image. The file can be narrowed down and displayed.

  FIG. 4 is a flowchart showing the flow of the sensitivity expression determination process in the present embodiment. The processing shown in FIG. 4 is executed by the control device 103 as a program that starts when the user instructs the user to associate the emotional expression of the main subject with the image file.

  In step S10, as described above, the control device 103 refers to the header information of the image file to determine whether or not the shooting mode is the portrait mode, thereby determining whether or not the main subject is a person. Judging.

  If it is determined in step S10 that the main subject is a person, the process proceeds to step S20. In step S20, as described above, the control device 103 performs scoring for determining the sensitivity expression of the main subject based on the smile level, the pose, the background color and brightness, and the process proceeds to step S40. .

  On the other hand, if it is determined in step S10 that the main subject is other than a person, the process proceeds to step S30. In step S30, as described above, the control device 103 performs scoring for determining the emotional expression of the main subject based on the color and brightness of the image in the AF area, and the complexity of the image. Proceed to S40.

  In step S40, the control device 103 determines the emotional expression of the main subject based on the result of scoring in step S20 or step S30. For example, as described above, when the total score is equal to or greater than a predetermined threshold, it is determined that the emotional expression of the main subject of the image is “fun”. On the other hand, when the total score is less than the predetermined threshold, it is determined that the sensitivity expression of the main subject of the image is “serious”. Thereafter, the process proceeds to step S50.

  In step S50, the control apparatus 103 records the determination result in step S40 in the header information of the image file, and proceeds to step S60. In step S <b> 60, the control device 103 determines whether processing for all image files recorded in the HDD 103 has been completed. If a negative determination is made in step S60, the process returns to step S10, and the process is repeated for an unprocessed image file. On the other hand, when an affirmative determination is made in step S60, the process ends.

  FIG. 5 is a flowchart showing the flow of image file search processing in the present embodiment. The process shown in FIG. 5 is executed by the control device 103 as a program that is activated when a user designates a search condition on the search condition designation screen shown in FIG. 2 and presses the “search” button 2a.

  In step S110, the control device 103 is designated as an image file that matches the search condition designated on the search condition designation screen from the image files recorded in the HDD 104, that is, as a search condition on the search condition designation screen. The sensibility expression is searched for the image file recorded in the header information as the sensibility expression of the main subject.

  Thereafter, the process proceeds to step S120, and the control device 103 displays a list of images of the image file extracted by the search process in step S110 on the monitor 105, and ends the process.

According to the present embodiment described above, the following operational effects can be obtained.
(1) The control device 103 identifies a main subject in the image, analyzes the image, detects a plurality of pieces of information for specifying the emotional expression of the main subject, for example, the degree of smile of a person, and the detected information Based on this, the sensitivity expression of the main subject was determined. This makes it possible to determine the sensitivity expression of the main subject in the image with high accuracy.

(2) The control device 103 determines whether or not the main subject is a person. If it is determined that the main subject is a person, the degree of smile of the person, the pose of the person, the background color, and the background If the main subject is determined not to be a person, scoring should be based on the background color, background brightness, and image complexity. I made it. Thus, the sensitivity expression of the main subject can be determined with high accuracy based on a plurality of factors for determining the sensitivity expression of the main subject.

(3) The control device 103 records the determined sensitivity expression of the main subject in association with the image file. As a result, by referring to the header information of the image file, it is possible to use the determination result of the sensitivity expression of the main subject.

(4) The control device 103 searches for image files of emotional expressions that match the conditions specified by the user on the search condition specification screen, and displays the search results as a list. Thus, the user can narrow down the image file based on the sensitivity expression of the main subject.

(5) The control device 103 performs the sensitivity expression determination process for the main subject when the execution of the process is instructed by the user. As a result, the user can instruct execution of processing at a desired timing.

-Modification-
The image processing apparatus according to the above-described embodiment can be modified as follows.
(1) In the above-described embodiment, the image processing apparatus 100 is, for example, a personal computer, and the control apparatus 103 performs the sensitivity expression determination process for the main subject shown in FIG. 4 at the timing instructed by the user. An example of execution has been described. However, the present invention is also applicable to other devices such as cameras. When applied to a camera, the main subject's sensitivity expression determination processing may be executed when an instruction from the user is given on the camera, or the main subject at the timing when the image file is generated by shooting. May be executed to write the sensitivity expression of the main subject in the header information of the image file. As a result, after the image file is copied from the camera to another device, the sensitivity expression of the main subject recorded in the header information of the image file can be obtained without performing the sensitivity expression determination processing of the main subject in the copy destination device. Can be used.

(2) In the above-described embodiment, an example in which the control device 103 identifies the subject included in the AF area as the main subject when the photographed image is photographed in a photographing mode other than the portrait mode. However, it is also possible to determine the subject included in the predetermined range including the AF area as the main subject by determining that the main subject exists in the AF area and a predetermined range outside the AF area.

(3) In the embodiment described above, an example in which the control device 103 determines whether or not the main subject is a person by determining whether or not the shooting mode at the time of shooting is the portrait mode will be described. did. However, subject recognition processing for recognizing the type of subject may be performed to determine whether or not the main subject is a person.

(4) In the embodiment described above, the control device 103 records the determination result of the sensitivity expression of the main subject in the header information of the image file of the captured image, thereby determining the determination result of the sensitivity expression of the image file and the main subject. An example of associating with was explained. However, the control device 103 may associate the image file with the determination result of the sensitivity expression of the main subject by other methods. For example, the control device 103 may associate the file name of the image file of the captured image with the determination result and record it as a separate file.

(5) In the above-described embodiment, only the image file in which the emotional expression matching the emotional expression designated on the search condition designation screen is recorded in the header information from among the image files recorded in the HDD 104. The example which extracted and displayed them as a list was demonstrated. However, the scoring result is recorded in the header information of the image file together with the information related to the emotional expression of the main subject, and the control device 103 arranges the images in order from the image with the highest score when displaying the list of images. You may change. Alternatively, when “fun” is designated as a search condition by the user, the control apparatus 103 displays a list of images of all the image files recorded in the HDD 104 as shown in FIG. The display order may be rearranged in descending order of the score. That is, you may make it display in an order from a thing with high compatibility with the sensitivity expression of the main subject designated as search conditions. As a result, rearrangement can be performed so that images that are likely to be viewed by the user are arranged in order from the top.

(6) In the above-described embodiment, an example has been described in which all the image files recorded in the HDD 104 are subjected to determination and recording of the sensibility expression of the main subject, or search for an image file. However, not all image files recorded in the HDD 104 but only image files in a specific folder existing in the HDD 104 may be targeted, or only image files selected by the user may be targeted.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired. Moreover, it is good also as a structure which combined the above-mentioned embodiment and a some modification.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus, 101 Operation member, 102 Connection IF, 103 Control apparatus, 104 HDD, 105 Monitor

Claims (7)

A main subject identifying means for identifying a main subject in the image;
An information detecting means for analyzing the image and detecting a plurality of pieces of information relating to a shape or color for specifying a sensitivity expression of the main subject;
A sensitivity expression determination means for determining a sensitivity expression of the main subject based on the plurality of pieces of information detected by the information detection means ;
Main subject determining means for determining whether or not the main subject is a person,
When the main subject determination unit determines that the main subject is a person, the information detection unit includes information on each of the person's smile level, person's pose, background color, and background brightness. When the plurality of pieces of information are detected and the main subject determination unit determines that the main subject is not a person, information on each of the background color, background brightness, and image complexity is stored in the plurality of pieces of information. An image processing apparatus characterized in that it is detected as information . A main subject identifying means for identifying a main subject in the image;
An information detecting means for analyzing the image and detecting a plurality of pieces of information relating to a shape or color for specifying a sensitivity expression of the main subject;
A sensitivity expression determination means for determining a sensitivity expression of the main subject based on the plurality of pieces of information detected by the information detection means;
Main subject determining means for determining whether or not the main subject is a person,
When the main subject determination unit determines that the main subject is a person, the information detection unit detects information about each of a person's smile level and a person's pose as the plurality of pieces of information. An image processing apparatus, wherein when the main subject determining means determines that the main subject is not a person, information relating to a background color and a background brightness is detected as the plurality of pieces of information. The image processing apparatus according to claim 1 or 2,
An image processing apparatus, further comprising: a recording control unit that records a determination result by the sensitivity expression determination unit in association with an image. The image processing apparatus according to claim 3.
A designation means for designating a sensitivity expression of a main subject as an image search condition;
Search means for searching for an image that matches the search condition specified by the specification means;
An image processing apparatus further comprising display control means for displaying a search result obtained by the search means. The image processing apparatus according to claim 4.
The image processing apparatus, wherein the display control means rearranges images and displays a search result based on a degree of matching with the sensitivity expression of the main subject specified as a search condition. In the image processing device according to any one of claims 1 to 5,
The image processing apparatus according to claim 1, wherein the main subject specifying unit, the information detecting unit, and the sensitivity expression determining unit perform processing when an image is taken. In the image processing apparatus according to any one of claims 1 to 6,
The image processing apparatus according to claim 1, wherein the main subject specifying unit, the information detecting unit, and the sensitivity expression determining unit perform the process when an instruction to execute the process is given by a user.

Images