JP2009087003A - Image retrieval device and image retrieval method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image retrieval device capable of shortening a retrieval time and of improving retrieval accuracy. <P>SOLUTION: This image retrieval device is configured to automatically retrieve image data matched with set retrieval conditions by a first CPU 28 and a second CPU 29 from among a retrieval object image data group recorded in a recording medium 15, and to display an image corresponding to the retrieved image data on a TFT panel 8 or the like, and to set an identification code for identifying whether or not the displayed image is a desired image by a first CPU 28 according to an operation from a key matrix 25, and to record the set identification code into a recording medium 15 by associating it with the image data corresponding to the displayed image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image search apparatus and an image search method for searching for desired image data from a search target image data group.

  In recent years, with the widespread use of digital cameras and digital video cameras, the amount of image data has increased dramatically, and the development of a technique that can quickly and accurately search for a target image is desired.

  As a method for searching for such an image, the following two types of methods are known.

  In the first method, a keyword reflecting the content of an image is assigned to the image by a human in advance, and an image to which a keyword equivalent to the keyword input by the user is added is extracted from the image database at the time of search. And present it.

  The second method is a method in which physical features such as colors, shapes, and textures included in an image are quantified and set as attribute values, and the information processing apparatus automatically searches using the attribute values. That is, in this method, each attribute value of the reference image is compared with each attribute value of each image of the search target image group stored in the image database, and an image with high similarity is extracted from the image database. Present as a search result.

  However, in the first method described above, if the user who is trying to search and the person who assigned the keyword to each image in the image database are different people, There is a case where the reference keyword does not match the keyword used in the image database, which causes a problem of omission of search.

  In the above-described second method, when attribute values are compared based on a predetermined algorithm and an image extracted as having a high degree of similarity is always determined to be highly similar to a reference image by human vision. However, there are many cases where similarity is rather low in terms of sensation, and the problem of low search accuracy has been pointed out.

As a technique for solving these problems, for example, in Japanese Patent Application Laid-Open No. 10-289240, a search is executed using a keyword assigned to an image, and a similarity search is executed using the attribute value of the image of the search result. A method has been proposed.
Japanese Patent Laid-Open No. 10-289240

  However, the technique described in the above Japanese Patent Laid-Open No. 10-289240 first searches for an image using a keyword, but the attribute value of the image may differ greatly even for images having the same keyword. Therefore, even if a similar image is subsequently searched based on the attribute value, it is not always possible to solve the decrease in search accuracy.

  Therefore, there is still a need for a technique that can shorten the search time and improve the search accuracy.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image search apparatus and an image search method capable of shortening search time and improving search accuracy.

  In order to achieve the above object, an image search apparatus according to a first invention corresponds to a search unit for automatically searching image data from a search target image data group, and image data automatically searched by the search unit. A display unit for displaying an image, an identification code setting unit for setting an identification code for identifying whether or not the image displayed on the display unit is a desired image, and the identification code displayed on the display unit And a recording unit for recording in association with image data corresponding to the recorded image.

  An image search device according to a second invention further includes an operation unit for correcting the identification code corresponding to the image displayed on the display unit by a manual operation in the image search device according to the first invention. The identification code setting unit sets the identification code corrected by the operation unit.

  Furthermore, in the image search device according to the third invention, in the image search device according to the second invention, the search unit automatically adds an identification code indicating that the image is a desired image to the image data obtained as a result of the automatic search. The operation unit can perform an operation of correcting the identification code corresponding to the image displayed on the display unit to an identification code indicating that the image is not a desired image.

  An image search device according to a fourth invention is the image search device according to the third invention, wherein the operation section includes an operation button that can be pressed, and the operation button is an erasure for erasing the image. It is also used as a button.

  An image search device according to a fifth invention is the image search device according to the fourth invention, wherein the operation for correcting the identification code corresponding to the image displayed on the display unit to an identification code indicating that the image is not a desired image, This is an operation of continuously pressing the erase button for a predetermined time.

  According to a sixth aspect of the present invention, there is provided an image search method for automatically searching image data from a group of search target image data, displaying an image corresponding to the automatically searched image data on a display unit, and displaying the image displayed on the display unit. Is a method for setting an identification code for identifying whether or not the image is a desired image and recording the identification code in association with image data corresponding to the image displayed on the display unit.

  According to the image search device and the image search method of the present invention, it is possible to shorten the search time and improve the search accuracy.

  Embodiments of the present invention will be described below with reference to the drawings.

[Embodiment 1]
1 to 3 show Embodiment 1 of the present invention, and FIG. 1 is a block diagram showing the configuration of a camera to which an image search apparatus is applied.

  As shown in FIG. 1, this camera includes a lens 1, a CCD 2, an imaging circuit 3, an A / D converter 4, a frame memory 5, a FIFO memory 6, a TFT liquid crystal driving circuit 7, and a TFT panel. 8, a backlight unit 9, a video output circuit 11, a video output terminal 12, a recording buffer 13, a recording medium interface (recording medium I / F) 14, a recording medium 15, an actuator driving circuit 16, Actuator 17, LCD display circuit 18, LCD panel 19, battery 21, backup power supply 22, power supply circuit 23, battery state detection circuit 24, key matrix 25, external wired data interface (external wired data I / F) 26, EEPROM 27, first CPU 28, and second CPU 29.

  The lens 1 is for forming an optical subject image on the CCD 2 and is a part of the imaging unit.

  The CCD 2 is an imaging device that photoelectrically converts an optical subject image formed by the lens 1 and outputs an electrical image signal, and is a part of the imaging unit. Here, a CCD is taken as an example of the image pickup device, but of course, a CMOS or other type of image pickup device may be used.

  The imaging circuit 3 performs various analog signal processing on the output from the CCD 2 and is a part of the imaging unit.

  The A / D converter 4 is for converting an analog image signal output from the imaging circuit 3 into a digital image signal (image data representing an image to be searched), and is a part of the imaging unit. is there.

  The frame memory 5 is a memory that stores an image signal of a moving image output in units of frames when displaying a through image, and further stores the still image when a still image is captured. Has been.

  The FIFO memory 6 is a memory for buffering when displaying or outputting an image.

  The TFT liquid crystal drive circuit 7 controls the TFT panel 8 based on the image signal output from the FIFO memory 6 and is a part of the display unit.

  The TFT panel 8 is capable of color display, and is used for displaying an image, various types of information relating to the camera, and the like based on the control of the TFT liquid crystal driving circuit 7, and is a part of the display unit. It is.

  The backlight unit 9 is provided on the back side of the TFT panel 8, is for illuminating the TFT panel 8 from the back side, and is a part of the display unit.

  The video output circuit 11 is for outputting the image signal from the FIFO memory 6 to the outside via the video output terminal 12.

  The video output terminal 12 is a connection terminal for connecting an external video cable or the like.

  The recording buffer 13 buffers when the image signal stored in the frame memory 5 is recorded on the recording medium 15, and constitutes a recording unit. The recording buffer 13 also performs buffering of signals read from the recording medium 15.

  The recording medium interface 14 is for controlling recording of data on the recording medium 15 and reading of data from the recording medium 15, and constitutes a recording unit.

  The recording medium 15 is a non-volatile recording medium (recording unit) for recording an image data group to be searched for images and other various data, and is configured to be detachable from the camera, for example. It has become.

  The actuator drive circuit 16 controls and drives the actuator 17 based on the control of the first CPU 28.

  The actuator 17 is a drive source for driving the lens 1 to perform autofocus, or to perform automatic zoom when an automatic zoom function is provided.

  The LCD display circuit 18 is a circuit for controlling the LCD panel 19 to perform various displays based on the control of the first CPU 28.

  The LCD panel 19 is configured to include, for example, a monochrome LCD, and the operation mode such as the shooting mode set in the camera, the number of images that can be recorded on the recording medium 15, the shutter speed and aperture value at the time of shooting. It is a display part for displaying information, such as.

  The battery 21 is the main power source of this camera.

  The backup power source 22 is a power source for supplying power for date display in this camera.

  The power supply circuit 23 is a circuit that performs control to supply power from the battery 21 and the backup power supply 22 to each circuit in the camera based on a command from the first CPU 28.

  The battery state detection circuit 24 detects the voltage of the battery 21 and the like, calculates the remaining battery capacity and the like of the battery 21, and outputs the result to the first CPU 28.

  The key matrix 25 is an operation unit and is a general term for operation input devices including various operation switches and operation buttons provided in the camera. As a specific example, the key matrix 25 is a power switch for turning on the power of the camera, a release button for inputting a photographing operation, a mode button for setting a photographing mode of the camera, and an image. For example, a frame change button for performing frame advance of a display image, an interruption button for interrupting an operation being executed, and the like. Of these, an erase button, a frame change button, and an interrupt button are operation buttons used when performing an image erase operation, and constitute an image erase operation unit 31 included in the key matrix 25. The image erasing operation unit 31 is not only used when erasing an image, but is also used as an operation unit when determining whether or not the automatic search result is correct after the image is automatically searched. Yes. A signal generated by operating the key matrix 25 is input to the first CPU 28.

  The external wired data interface 26 is configured by, for example, a USB or the like, and is an interface for transmitting / receiving data to / from an external device.

  The EEPROM 27 is a non-volatile storage medium for storing various data relating to the camera, a processing program executed by the first CPU 28 or the second CPU 29, and a rewritable storage medium such as a flash ROM (Flash ROM), for example. It is configured as. The EEPROM 27 further stores feature data for personal authentication, subject classification information, and face image data as will be described later.

  The first CPU 28 is a main CPU, and is a control unit for comprehensively controlling each circuit of the camera. The first CPU 28 has a system control function for controlling the system of the camera. The system control function includes an image search / classification function, a personal authentication function, and a clock function. Thus, the first CPU 28 sets an identification code setting unit (automatic identification code setting unit, manual identification code setting unit) for setting an identification code for identifying whether or not the image is a desired image according to the search condition. It has also become a thing.

  Here, the clock function is a function for measuring time similar to that provided in a normal computer or the like, and is a function used for managing the shooting date / time of an image, the creation date / time of an image file, and the like. Note that other functions of the first CPU 28 will be described in detail later.

  Thus, the first CPU 28 mainly performs control, and the second CPU 29 mainly performs processing for handling image data.

  That is, the second CPU 29 processes image data stored in the frame memory 5. The second CPU 29 has an image analysis function, a feature point extraction function, an image compression / decompression function, and a recording medium access function.

  The image compression / decompression function is a function of, for example, JPEG compression or the like on the image data stored in the frame memory 5 or decompressing the compressed image data read from the recording medium 15.

  The recording medium access function is a function for controlling access to the recording medium 15 by the recording medium interface 14.

  Note that other functions of the second CPU 29 will be described in detail later. The first CPU 28 and the second CPU 29 constitute a search unit that searches the search target image data group recorded on the recording medium 15 for image data that matches the set search conditions.

  Such an image search apparatus first recognizes a subject (such as a person's face) included in the image data recorded on the recording medium 15 by an image analysis function (such as a face recognition function), and recognizes the characteristics of the recognized subject. A point is extracted by a feature point extraction function, and the extracted feature point is registered as feature data for personal authentication in the EEPROM 27, and face image data to be recognized is recorded in the EEPROM 27 in the form of, for example, a thumbnail image. Link information of the data and face image data and a flag (a flag indicating that the person is a registered person) is recorded in the subject classification information.

  If registration based on such automatic search is performed, then the result of automatic registration is manually verified, and the image data with an inappropriate flag is cleared. Can be done.

  Therefore, first, an operation at the time of subject registration will be described with reference to FIG. FIG. 2 is a flowchart showing the operation when the subject is automatically searched and registered by the image search device.

  When the camera is set to the subject registration mode, this process is started. When the process is started, first, the internal counter n is set to 1 (step S1).

  Next, from the images recorded on the recording medium 15, the nth image is reproduced and displayed on the TFT panel 8 or the like (step S2).

  Subsequently, the image being reproduced is analyzed by the image analysis function of the second CPU 29, and a face included in the image is detected (step S3).

  Then, the first CPU 28 determines whether or not a face is detected in the image (step S4). If it is determined that the face is detected, the feature point extraction function of the second CPU 29 analyzes the detected feature point of the face. To extract (step S5).

  Next, by comparing the feature points of the face obtained by the analysis with the feature points of the face of the registered person, it is determined whether or not the face is the face of the already registered person. The determination is made by the personal authentication function of the first CPU 28 (step S6).

  If it is determined that the face is a person who has already been registered, an identification code is added to the flag of an existing person photo (a flag indicating that the person is shown) in the file header of the current target image. (Step S7).

  If it is determined in step S6 that the face of the person is not registered, the face feature point data is recorded in the personal authentication feature data portion of the EEPROM 27 as a new person and the face to be recognized. The image data is recorded in the form of, for example, a thumbnail image, and the feature point data and the link information between the face image data and the flag are recorded in the subject classification information portion (step S8).

  Then, a flag indicating a newly registered person is added to the file header of the currently targeted image (step S9).

  If a flag is added to the current target image by completing the process of step S7 or step S9, it is next determined whether or not there is another face in the current target image (step S10). Here, since a plurality of faces may be detected in one image, the processing in steps S5 to S9 is performed for each face included in one image.

  If it is determined in step S10 that there is another face, the process returns to step S5 to perform the process described above for the next face.

  On the other hand, if it is determined in step S10 that there is no other face, or if it is determined in step S4 that no face has been detected in the image, whether or not the analysis of all the images included in the recording medium 15 has been completed. Is determined (step S11).

  Here, when it is determined that the analysis of all the images included in the recording medium 15 has not been completed yet, the counter n is incremented (step S12), the process returns to step S2, and the next image is described above. Perform the following process.

  If it is determined in step S11 that the analysis of all the images included in the recording medium 15 has been completed, the subject registration process is terminated.

  Next, search result determination processing executed after subject registration as described above is performed on image data recorded on the recording medium 15 will be described with reference to FIG. FIG. 3 is a flowchart showing a search result determination process in the image search apparatus.

  When this processing is started, first, the registered face image data recorded in the EEPROM 27 is displayed on the TFT panel 8 or the like, and waits for the face to be judged to be selected from the displayed faces (step) S21).

  When a face is selected, the flag corresponding to the selected face is read by referring to the subject classification information (step S22), and an image corresponding to the flag is further searched from the recording medium 15. Then, it is listed up (step S23).

  After that, it is determined whether or not there is a target image with a flag corresponding to the selected face (step S24). If it is determined that there is no target image, for example, “There is no image corresponding to the selection. ”Or the like (step S25), the process returns to step S21 to wait for another face to be selected.

  If it is determined in step S24 that there is a target image, the listed target image is reproduced and displayed on the TFT panel 8 or the like (step S26).

  Then, the first CPU 28 determines whether or not the delete button of the image delete operation unit 31 has been pressed (step S27). If it is determined that it has been pressed, an identification error display or the like is performed to clear the image flag (step S27). S28). The identification error is displayed by displaying information such as "There was an error in automatic search". Instead, for example, information such as "Image was removed from the target list" is displayed. It does not matter if you do.

  If it is determined in step S27 that the delete button has not been pressed, or if the processing in step S28 has been completed, it is next determined whether or not the frame change button of the image delete operation unit 31 has been pressed (step S27). Step S29).

  Here, if it is determined that the frame change button has been pressed, it is further determined whether or not the reproduction display of all target images has been completed (step S31).

  If it is determined that the reproduction display of all target images has not been completed yet, the next target image is reproduced and displayed (step S32), and the process returns to step S27 to determine the delete button. Do as follows.

  On the other hand, if it is determined in step S29 that the frame change button has not been pressed, it is further determined whether or not the interrupt button has been pressed (step S30).

  If it is determined that the interrupt button has not been pressed, the process returns to step S27 to determine the delete button as described above.

  If it is determined in step S30 that the interrupt button has been pressed, or if it is determined in step S31 that all the target images have been reproduced and displayed, the search result determination operation is terminated (step S33). ), This process is terminated.

  In this way, after automatic registration and further manual registration correction, an inappropriate image flag that does not match the search condition is cleared, so the image is more efficiently processed. You will be able to search.

  Here, the image search is performed as follows as an example. First, a subject (such as a human face) to be searched is imaged. Then, the subject included in the captured image data is recognized by the image analysis function, and the feature point of the recognized subject is extracted by the feature point extraction function. Further, it is determined whether or not the feature point of the subject to be searched matches any of the feature points of the subject already registered using the personal authentication function. Here, if they match, images containing the corresponding subject are listed by the image search / classification function using flags and displayed in the order listed. At this time, an inappropriate image is not listed because the flag is cleared.

  In the above description, an automatic search is performed according to the set search condition, and an identification flag is temporarily assigned to image data determined to be applicable as a result of the automatic search, and then the identification flag is manually corrected. However, it is not limited to this. For example, even if the identification flag is manually corrected, if the automatic search is performed again (subject registration processing in the above-described example), the identification flag may be rewritten to the result of the automatic search. Therefore, the identification flag added by the automatic search is not corrected, and separately from this, an identification flag indicating whether or not the target image is a desired image may be added separately.

  According to the first embodiment, as a result of performing an automatic search by setting a search condition so that a desired image is searched, an image that is not a desired image is included in the obtained image. Even so, since it can be corrected manually, it is possible to improve the subsequent search accuracy and search speed.

  In addition, when manually correcting, since the deletion button that is usually provided in an apparatus that handles images, such as a camera, is also used, there is no need to provide a separate operation member, and “delete registration”. Since the operation is common in terms of “deleting the image” and “deleting”, subjectively unified and easy-to-understand operability can be realized.

[Embodiment 2]
FIG. 4 shows a second embodiment of the present invention, and is a flowchart showing search result determination processing in the image search apparatus. In the second embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals, description thereof is omitted, and only different points will be mainly described.

  Also in the second embodiment, the configuration of a camera to which the image search apparatus as shown in FIG. 1 is applied is assumed. The process for registering the subject recorded on the recording medium 15 is the same as the process shown in FIG.

  Next, search result determination processing according to the second embodiment will be described with reference to FIG. In FIG. 4, the same reference numerals are given to the portions that perform the same processing as in FIG. 3 of the first embodiment described above.

  When this process is started, the processes of steps S21 to S27 are performed as described above.

  Subsequently, when it is determined in step S27 that the delete button has been pressed, time measurement is started (step S41), and the process waits for the delete button to be released (step S42).

  Thereafter, when it is determined that the delete button has been released, the time measurement is ended (step S43). Then, based on the measured time, it is determined whether or not the delete button has been pressed for a predetermined time, for example, 1 second or more (step S44). Here, when it is not pressed for 1 second or more, the process proceeds to step S29.

  On the other hand, if it is determined in step S44 that the delete button has been pressed for 1 second or longer, the process proceeds to step S28 to display an identification error, clear the image flag, and then the process proceeds to step S29.

  Other processing is almost the same as the processing shown in FIG. 3 of the first embodiment.

  According to the second embodiment, the same effect as that of the first embodiment described above can be obtained, and the image displayed on the display unit can be handled only when the delete button is continuously pressed for a predetermined time or longer. Since the identification code is erased, the identification code is not erased even if the erasure button is momentarily pressed without intention, and erroneous operation or the like can be prevented.

  In each of the above-described embodiments, as an example of search, a case is described in which an image is analyzed to extract feature points of the face image portion, and image search is performed based on the extracted feature points. The present invention is not limited, and the present invention can be similarly applied to other searches such as a search based on a keyword related to an image and a search based on a physical feature of an image.

  The present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, you may delete a some component from all the components shown by embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined. Thus, it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

[Appendix]
According to the above-described embodiment of the present invention described in detail above, the following configuration can be obtained.

(1) a search unit that automatically determines whether each image data included in the search target image data group satisfies a predetermined search condition;
An automatic identification code setting unit that automatically adds an identification code indicating that the image data meets the predetermined search condition to the image data determined to satisfy the predetermined search condition;
A display unit for displaying an image corresponding to the image data determined to satisfy the predetermined search condition;
A manual identification code setting unit for manually correcting the identification code of the image displayed on the display unit to an identification code indicating that the image data does not satisfy the predetermined search condition;
An image search apparatus comprising:

1 is a block diagram illustrating a configuration of a camera to which an image search device according to a first embodiment of the present invention is applied. 5 is a flowchart showing an operation when a subject is automatically searched and registered by the image search apparatus according to the first embodiment. 5 is a flowchart showing search result determination processing in the image search apparatus according to the first embodiment. 9 is a flowchart showing search result determination processing in the image search apparatus according to the second embodiment of the present invention.

Explanation of symbols

1 ... Lens 2 ... CCD
DESCRIPTION OF SYMBOLS 3 ... Imaging circuit 4 ... A / D converter 5 ... Frame memory 6 ... FIFO memory 7 ... TFT liquid crystal drive circuit (display part)
8 ... TFT panel (display unit)
9 ... Backlight unit (display unit)
DESCRIPTION OF SYMBOLS 11 ... Video output circuit 12 ... Video output terminal 13 ... Recording buffer (recording part)
14. Recording medium interface (recording medium I / F) (recording unit)
15. Recording medium (recording unit)
DESCRIPTION OF SYMBOLS 16 ... Actuator drive circuit 17 ... Actuator 18 ... LCD display circuit 19 ... LCD panel 21 ... Battery 22 ... Backup power supply 23 ... Power supply circuit 24 ... Battery state detection circuit 25 ... Key matrix (operation part)
26 ... External wired data interface (external wired data I / F)
27… EEPROM
28 ... 1st CPU (search part, identification code setting part)
29 ... 2nd CPU (search part)
31. Image erasing operation section (including operation buttons and erasing buttons)

Claims (6)

A search unit for automatically searching for image data from the search target image data group;
A display unit for displaying an image corresponding to the image data automatically searched by the search unit;
An identification code setting unit for setting an identification code for identifying whether or not the image displayed on the display unit is a desired image;
A recording unit for recording the identification code in association with image data corresponding to the image displayed on the display unit;
An image search apparatus comprising: An operation unit for manually correcting the identification code corresponding to the image displayed on the display unit;
The image search apparatus according to claim 1, wherein the identification code setting unit sets an identification code corrected by the operation unit. The search unit automatically sets an identification code indicating that the image is a desired image in the image data obtained as a result of the automatic search,
The operation unit can perform an operation of correcting the identification code corresponding to the image displayed on the display unit to an identification code indicating that the image is not a desired image. Image search device.   The image search apparatus according to claim 3, wherein the operation unit includes an operation button that can be pressed, and the operation button also serves as an erase button for erasing an image. .   The operation of correcting the identification code corresponding to the image displayed on the display unit to an identification code indicating that the image is not a desired image is an operation of continuously pressing the erasing button for a predetermined time. 5. The image search device according to 4. Automatically search for image data from the search target image data group,
The image corresponding to the automatically searched image data is displayed on the display unit,
Set an identification code for identifying whether the image displayed on the display unit is a desired image,
Recording the identification code in association with image data corresponding to the image displayed on the display unit;
An image search method characterized by that.

Images