JP2008098987A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus which is more user-friendly than a conventional one when an image is retrieved in reproducing photographed images. <P>SOLUTION: The apparatus has a characteristic part recognizing means for recognizing a characteristic part in a subject photographed image, and counts the number of characteristic parts recognized by the characteristic part recognizing means in recording photographed data, and records the quantity of the obtained characteristic parts in a header part of the photographed data. Furthermore, the apparatus retrieves and displays data based on the number of characteristic parts recorded in the header part of the photographed data in reproducing the photographed data. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a mobile device (mobile phone, PDA) or other imaging device with a photographing function, and more particularly to a feature recognition technique for a captured subject image and an image search / playback technology using the feature.

  In an imaging apparatus such as a digital camera, when a large-capacity recording medium is used, a function of searching and displaying a captured image at the time of reproduction becomes important. Conventionally, there has been a technique for retrieving and displaying a photographed image based on the photographing date and various header information. In these functions, it is very important to determine whether or not a shooting situation that is more likely to remain in the photographer's mind can be used as a search key.

As one of the techniques for extracting the characteristics of a photographed image, many techniques focus on the person in the photographed image. As a method for extracting a human face from a captured image, a method is known in which skin color data is extracted from an original image and a cluster of photometric points determined to be in the skin color range is recognized as a face.
This method is disclosed in Patent Document 1 and the like. Further, Patent Document 2 discloses a method for recognizing a face area by converting photometric data into hue and saturation, and creating and analyzing a two-dimensional histogram of the converted hue and saturation. Further, Patent Document 3 discloses a technique for extracting a face candidate area corresponding to the shape of a human face and determining the face area from the feature amount in the area. In addition, a method for extracting a face region by extracting the outline of a human face from an image and determining a face region or preparing a template having a plurality of face shapes and determining the correlation with the template. Has also been proposed.

  Further, regarding face recognition and eye detection, technologies and reference materials are introduced in detail in Patent Documents 4 and 5. In addition, Patent Document 6 discloses a device that detects red-eye and gold-eye portions in an image, counts the number of detected portions, displays the detected number in the image, and prevents oversight.

JP-A-52-156624 Japanese Patent Laid-Open No. 04-346333 Japanese Patent Laid-Open No. 08-063597 JP 09-251534 A JP-A-10-232934 JP 2004-208279 A

  Also in the present invention described later, face detection, person detection, eye detection, and non-human face detection are performed. In this detection process, an appropriate technique as described above can be used.

  In the conventional image retrieval techniques as described above, the elements for retrieval are not so strong that the target image is often not displayed smoothly.

  Therefore, in the imaging apparatus of the present invention, focusing on the face of the subject to be photographed in many cases, the number (including animal faces) and the number of persons are photographed as parameters indicating the characteristics of the image. At times, it is added to image data and used as a search condition to provide an imaging device that is easier to use than conventional ones.

  An imaging apparatus according to the present invention includes a subject imaging means, imaging data recording means, recorded data reproduction means, and feature portion recognition means for recognizing a feature portion in a subject captured image, and records imaging data. Sometimes, the number of feature portions recognized by the feature portion recognition means is counted, and the obtained number of feature portions is recorded in a header portion of the imaging data. Further, the present invention is characterized in that it is possible to search and display the number of the characteristic parts recorded in the header part of the imaging data as a search condition when reproducing the imaging data. As a result, it is possible to search for the number of features at the time of reproduction, improving usability.

  The number of faces in the subject can be used for the feature part. By recording the number of recognized faces and enabling search execution as a search condition, an appropriate search result as intended can be obtained. Alternatively, the number of pairs of eyes in the subject may be used for the feature portion, and similarly, an appropriate search result as intended can be obtained. In addition, the number of faces of the person in the subject can be used as the feature section, and the number of recognized faces of the person is recorded, and the search can be performed as a search condition to perform the search as intended.

  The imaging apparatus according to the present invention also includes a first display mode that displays only an image that matches the search condition with the specified number of features when reproducing captured image data, up to an image with a feature after a matching image priority display. At least two of a second display mode for displaying the image and a third display mode for displaying all the images after priority display of the matching image and the image having the characteristic portion. Equipped with either a display method that displays only those whose number of characteristic parts of the search condition is completely matched, a display method that displays even partial matches, or a display method that displays all images with priority. Make it even better.

  According to the imaging apparatus of the present invention, the number of feature parts of the captured image is counted and recorded in the header portion of the captured data, so that the search condition setting becomes appropriate and the intended captured data (image) is obtained. Search and display smoothly, and usability is greatly improved.

  In the present invention, as a technique for using the feature portion of the image, the number of extracted pattern feature portions such as the number of faces in the subject, the number of human faces, or the number of eye pairs is newly counted to obtain the captured image data. As described above, the number of characteristic portions (number of sets) is recorded in the header portion of the imaging data at the time of recording so that it can be used for retrieval at the time of reproduction. Some prior arts have the idea of counting features and displaying them on a monitor, but there has been no idea in the past to record the number of features as data and link this to image search operations. . Although there is a conventional technique for recognizing a person's face and performing various processes, there was no concept used for searching. By using the present invention, which is not seen in the past, the feature parts are aggregated into the number and recorded for search, the search conditions can be easily remembered and set easily, and the usability can be greatly improved. It becomes.

[First Embodiment]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a first embodiment of an imaging apparatus according to the present invention. FIG. 2 is a conceptual diagram of presetting for face recognition, and FIG. 3 is a conceptual diagram of presetting for display after search. 4 to 7 are flowcharts showing the operation of this embodiment.

  In FIG. 1, the optical unit (100) includes a lens (101), an aperture / shutter (102), an optical low-pass filter (103), and a CCD (104). The subject image input to the optical unit (100) is converted into an electrical signal by the CCD (104) and input to the image processing unit (200). The optical elements such as the lens (101) and the diaphragm (102) and the CCD (104) are driven via an optical system driver (105) and a CCD driver (106), respectively. The image processing unit (200) performs AE processing, white balance processing, and the like during imaging. The recorded image is temporarily stored in the image buffer memory (202) via the BUS line (201), and is sent via the BUS line (201), the compression / decompression processing unit (203), and the image recording I / F unit (204). Finally, it is stored as a JPEG image in a recording medium (205) such as a memory card or HDD. The image processing unit (200) processes the input signal from the CCD (104) for monitoring during monitoring, and reads the read data from the recording medium (101) via the compression / decompression processing unit (203) during image reproduction. Then, it is processed for reproduction, output to the display unit (LCD panel) (206), and displayed.

  The image buffer unit (202) is a part for temporarily storing data before and after image processing, so that the shooting interval is not extended due to the influence of a writing system that is slow in processing particularly in continuous shooting. , With the ability to temporarily store several images at the highest image quality. The operation unit (300) shows a release SW, a zoom SW, a shooting mode setting SW, a menu SW, a search SW, etc., and a push button type, a lever type, and a menu display & selection & determination button using an LCD panel. Any SW type such as a type can be used, and an input result is input to the control / arithmetic unit, thereby triggering the control / arithmetic unit to perform a predetermined operation. A control operation unit (400) provided with a nonvolatile memory element for setting storage controls the above-described units.

  In the configuration of the first embodiment of the present invention, in the configuration described above, as a prior setting, the face recognition setting and the search setting are performed in the flow as shown in FIG. 2 and FIG. Shall be retained. That is, for the face recognition setting, the menu button is pressed from the normal monitoring mode before shooting, various setting items are read, and the face recognition setting items are displayed. Thereafter, the setting of each item is selected as necessary in the flow as shown in (DP01) to (DP03) in FIG. In the present embodiment, face recognition setting → present (always) → face recognition post-processing setting → count number display setting → selected is selected and determined. After setting, the menu button is pressed again to return to the normal monitoring mode (DP04). In this state, the face counting function is enabled, the face part in the subject being monitored is recognized, and the area recognized as a human face at all times (at predetermined intervals) as shown in (DP05) of FIG. The number is displayed. Note that the items set in this pre-configuration are stored in a non-volatile storage element such as EEPROM even when the camera is turned off. Settings are maintained. The setting state is maintained in the same manner for the search setting at the time of image reproduction, which will be described later.

  Subsequently, an imaging procedure will be described. In the first embodiment, a description will be given assuming that the presetting as described above is performed in advance. When the camera is activated (powered on) while the shooting mode button is set to the shooting mode, the camera enters a shooting preparation (subject monitoring) state. In this state, the imaging is continued, and the liquid crystal monitor is replaced with the latest image at a predetermined interval. In parallel with this, a feature portion extraction process for detecting a human face portion at a predetermined interval is performed.

  In this case, as a method for extracting a human face from a photographed image, various methods are already known, such as extracting skin color data from an original image and recognizing a cluster of photometry points determined to be a skin color range as a face. ing. That is, as already mentioned, techniques disclosed in Patent Documents 1 to 5 are widely known as this kind of technique. With regard to these technologies, each patent document introduces technologies and reference materials in detail, and in the present invention, face detection, person detection, eye detection, and non-human face detection processes themselves are the essence of the invention. Is not related to the above description, and the description thereof is omitted here. Any of the above-described techniques may be used for the face detection process in the present invention, and any appropriate new technique other than those exemplified may be used.

  Next, the feature extraction operation in the present embodiment will be described along the flowcharts shown in FIGS. In a normal shooting preparation state (subject monitoring mode), image capture (S101) and image processing are performed (S102), monitor thinning processing is performed (S103), and an image is displayed on the display unit (206). (S112). Here, when the constant (face recognition) mode is set (S104: YES), face recognition processing (S105) and the number of recognized faces (counting) are performed (S106). When the count number display (item) is set to display (S107: YES), when a face area of a person in the captured image is detected, a face area boundary line is displayed in a red frame during monitor display. Is displayed, and a count value (a value counted according to the number of red frames) is displayed around the red frame as needed (S108: face area count display composition; see FIG. 2). The count value display may be displayed for each frame, or the total count may be displayed on the end of the monitor. The red frame and the red frame count are not recorded as captured image data, but face count data is recorded as the person count data for the captured image in the header portion of the image data.

  In this state in the monitoring mode, the release press is monitored (S109). When half-press (first stage ON) is detected (S109: YES), “count value (face count)” is temporarily stored (S110), and the process proceeds to the release process shown in FIG. On the other hand, if there is no release operation (S109: NO), after waiting for a fixed time (S111), it will return to step; S101.

  In the release process (FIG. 5), the photometry process (S201) and the distance measurement process (S202) are performed in response to the half-press operation, the lens extension (S203) is performed, and then the release operation is monitored (S204). ).

  If the release button is pressed halfway (S204: NO), a face recognition subroutine is performed (S300). In the face recognition subroutine, when face recognition (during shooting) is set (S301: YES), a captured image is captured (S302), predetermined image processing (S303) is performed, and a thinning process for monitoring is performed. (S304). Also, face recognition processing (S305) and the number of recognized faces (counting) are performed (S306). When the count number display (item) is set to display (S307: YES), the face area count display composition (S308; FIG. 2) is displayed on the display unit (206) in the same manner as described above (step; S108). See also). These processes may be performed in parallel with the release operation monitoring operation using an interrupt process or the like.

  In the release operation monitoring, when the release is continuously pushed in (second stage ON), that is, when photographing is performed (S204: YES), the photographed image is captured (S205), the shutter is closed, and the lens is closed. Returning to the specified position (S206), predetermined image processing (S207) and image compression processing are performed (S208). If face recognition is set (S209: YES), face count data is combined with the header to create an image data file (S210) and recorded on the recording medium (S212). If face recognition is not set Then, a file of only captured image data (no face count data is written) is created (S211) and recorded on a recording medium (S212). Here, as a configuration to save the image file in the EXIF format as the most commonly used EXIF-compliant JPEG image, the EXIF header is an area that can be freely set for each manufacturer (specified EXIF It is assumed that face count data is described in the free space portion of the format. Of course, if a setting area common to manufacturers is provided by new standardization or the like, that area may be used.

  In this way, it is assumed that about 100 is shot in the same shooting procedure. Assume that there are 10 subject persons, 5 persons, 5 persons, 5 persons, 2 persons, and 2 persons. When shooting is finished, the mode changes to the playback mode.

  In the playback mode of captured images, (pre-) search settings (change settings if already set) for searching for images to be displayed are possible. That is, in the search setting, in the playback mode, the menu button is pressed to read various setting items, and the search setting item is displayed from among them. Thereafter, the setting of each item is selected as necessary in the flow of (DP11) to (DP13) as shown in FIG. In the present embodiment, search condition selection → number of faces → reproduction display settings after search → only matching images are selected and determined in order. After setting, press the menu button again to return to the normal playback mode. This set item is stored in a non-volatile storage element such as EEPROM even when the camera is turned off, so once set, the setting is maintained until the setting is changed. .

  When searching for an image, the user presses a search button, inputs necessary items in accordance with preset search conditions, and executes the search. The image search procedure is shown in the flowchart of FIG. Here, since the setting condition is person face count (S402: YES), and only matching images are searched (S403: YES), it is assumed that the input of the search count is requested and two persons are input (S404). .

  Thereafter, a search based on the input face count “2” is performed (S405 to S407). As described above, the face count data used for the search is described in the EXIF header in an area that can be freely set for each manufacturer. The CPU searches the face count data part described in the header part of the image file recorded on the recording medium (S406), and creates and stores management information regarding the matched file group (S407). In addition, a search execution flag relating to the history of the search is set (S408), and the process proceeds to the reproduction mode. This flag may be reset according to various systems, such as when playback is finished, when the power is turned off, or when starting up, according to the setting of whether to leave a separate search history.

  Thereafter, as shown in the flowchart of FIG. 7, since (S501: YES) and (S502: NO), the image file is accessed based on the management information (S517), and the first matching image is displayed on the display means. It is displayed (S518). Then, when the return button is pressed in the same way as normal playback (S519, S520), the next matching file is called from the management information data and displayed on the display means (S517, S518). This is repeated in sequence, and the display of matching images is repeated so that only files in the management information are accessed.

  The management information may be stored in a temporary memory (such as a buffer memory), may be recorded in a non-volatile memory, or may be stored in a recording medium. Alternatively, the search execution history may be recorded, and if there is a history at the time of startup or the like, the file search is performed again, and the management information is recreated and stored again according to the set search conditions. Either way. Once the user performs a search according to the conditions set by himself / herself, only the files that meet the conditions can be displayed thereafter, and the usability of browsing the file especially in the case of a large-capacity recording medium is improved.

[Second Embodiment]
A second embodiment will be described. In the second embodiment, in FIG. 2 and FIG. 3, face recognition setting (during shooting), eye (number of pairs) count, no count number display, search setting, eye (number of sets) count, and fuzzy search are selected in advance I will explain it as if I had done it. In the following description of the control flow, the same flowcharts of FIGS. 4 to 7 as in the first embodiment are used.

  When the camera is activated (powered on) while the shooting mode button is set to the shooting mode, the camera enters a shooting preparation (subject monitoring) state.

  In this state, face detection is not performed in accordance with the setting condition (setting only for shooting). When the release button (half-press) is pressed, simple imaging is performed here, and eye detection (techniques such as the above-mentioned patent documents are used for pattern recognition and the like). Regarding the face recognition and eye detection in this case, the technologies and reference materials are introduced in detail in the above-mentioned Patent Document 4 and Patent Document 5. For example, there is a method of detecting a pattern of a black or other color (blue, green, etc.) round region having a white portion at a specific interval. Since the eye detection method in this embodiment is not related to the essence of the invention, any prior art may be used without any particular limitation. When an eye pattern that can be recognized as an eye pattern is detected in the captured image, the number of sets is counted, and the total number of counts is temporarily stored.

  After that, as in the first embodiment, photometry, distance measurement, lens drive, etc. are performed, and when the release button is pressed (fully pressed), imaging is performed, and after the image processing and compression, the temporary storage is performed. The total number of sets of eyes is recorded as total count data for the number of sets of eyes for the photographed image in the header portion of the image data (in the case of an EXIF-compliant JPEG image, the free space portion of the prescribed EXIF format).

  In this way, it is assumed that about 100 images have been captured by the same imaging method. Assume that there are 10 subjects in one set, 2 in 5 sets, 5 in 3 sets, 5 in 4 sets, and 2 in 4 sets or more. Shooting is completed and the mode switches to the playback mode.

  A search button is pressed, necessary items are input according to a preset search condition, and a search is executed (see a flowchart showing an image search procedure in FIG. 6). Here, it is assumed that the setting condition is an ambiguous search (S403: NO, S410: YES), and the search condition is the number of pairs of eyes, so that an input of the search count is requested and two sets are input ( S411).

  Thereafter, a search based on the input face count “2” is performed (S412 to S414). The CPU searches the face count data part described in the header part of the image file recorded on the recording medium, and creates and stores management information regarding the matched file group. Also, a search execution flag related to the history of the search is set (S408). This flag may be reset according to various systems, such as when playback is finished, when the power is turned off, or when starting up, according to the setting of whether to leave a separate search history. In this embodiment as well, the image file is stored in conformity with the most commonly used EXIF format. In the EXIF header, the face count is set in an area that can be freely set for each manufacturer. Data shall be described (if a standard area common to manufacturers is provided by new standardization, that area may be used).

  Thereafter, as shown in the flowchart of FIG. 7, since (S501: YES), (S502: YES), (S503: YES), the image file is accessed based on the management information (S504), and the first match Is displayed on the display means (S505). Then, as in normal playback, when the send / return button is pressed (S513, S514), the next matching file is called from the management information data and displayed on the display means (S504, S505). When the matching file ends (S506: YES), the search condition corresponding setting is an ambiguous setting, so a file close to the search condition (in this embodiment, an image with one set of eyes is continuously displayed (S507 to S511), After completion, three sets are displayed (S507 to S511), and then four or more sets are displayed (S507 to S511) When all the searched files are displayed (S509: YES), then Returns to the display of the two sets of images (the first image displayed after the search), and the following is repeated: The only difference from the search with only a match is that the image file where the eye was found is displayed in an expanded manner. This is different from the display of all files.

  The management information may be stored in a temporary memory (such as a buffer memory), may be recorded in a non-volatile memory, or may be stored in a recording medium. Alternatively, the search execution history may be recorded, and if there is a history at the time of startup or the like, the file search is performed again, and the management information is recreated and stored again according to the set search conditions. Either way. Once the user performs a search according to the conditions set by himself / herself, only the files that meet the conditions can be displayed thereafter, and the usability of browsing the file especially in the case of a large-capacity recording medium is improved.

  The present invention is not limited to the embodiment described above. For example, by using a face recognition function other than a person, counting the number of pairs of eyes, and the like, it becomes easy to search for an image showing yourself and your pet (this is the search for the number of faces and the number of sets of eyes). Can be executed by specifying 2). In addition, for face recognition functions other than humans, there are various recognition methods such as settings that allow skin colors other than skin color, and the number of eye pairs can be detected in almost the same way as humans. It is.

It is a block diagram which shows the structure of the imaging device which concerns on this invention. It is a conceptual diagram of the preset for face recognition. It is a conceptual diagram of presetting for display after search It is a flowchart which shows the imaging operation of embodiment. It is a flowchart (2) which shows the imaging operation of embodiment. It is a flowchart which shows search operation of embodiment. It is a flowchart which shows the reproduction | regeneration operation | movement of embodiment.

Explanation of symbols

(100)… Optical unit
(101)… Lens
(102)… Aperture / Shutter
(103) ... Optical low-pass filter
(104)… CCD
(105) ... Optical system driver
(106) ... CCD driver
(200) ... Image processing unit
(201)… BUS line
(202)… Image buffer memory
(203) ... Compression / decompression processor
(204) Image recording I / F section
(205) ... Recording media such as memory cards and HDDs
(206) ... Display (LCD panel)
(300)… operation unit
(400)… Control operation part

Claims (6)

  A subject imaging unit; an imaging data recording unit; a recorded data reproducing unit; and a feature recognition unit that recognizes a feature in the captured image of the subject. An image pickup apparatus that counts the number of recognized feature portions and records the obtained number of feature portions in a header portion of the image pickup data.   The imaging apparatus according to claim 1, wherein the imaging apparatus is configured to be able to search and display the number of the characteristic parts recorded in the header part of the imaging data as a search condition when reproducing the imaging data.   The imaging device according to claim 1, wherein the characteristic portion is the number of faces in the subject.   The imaging device according to claim 1, wherein the characteristic portion is the number of pairs of eyes in the subject.   The imaging device according to claim 1, wherein the characteristic portion is a number of faces of a person in the subject. A first display mode that displays only images that match the specified search condition for the number of features when reproducing captured data; and a second display mode that displays up to images with features after matching image priority display The image pickup apparatus according to claim 2, further comprising: at least two of a third display mode in which all images are displayed after priority display of a matching image and an image having a characteristic part.

Images