JP2010081528A - Image processing apparatus, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily and surely secure an image that enables a failed image to be complemented. <P>SOLUTION: When it is determined that a portion of a registered failed image belongs to a through-picture or that a feature quantity in capturing the through-picture is matched with a feature quantity of the failed image, a CPU 112 instructs a camera 10-B (or a sub imaging section 20) to start capturing a complementary image. When the camera 10-B receives the instruction via a communication I/F 76, thereafter, an image is periodically captured by an imaging section 102-B and captured images are sequentially stored in an image storage section 132-B as complementary images. This is also achieved by capturing a frame image of a moving picture or by consecutively photographing a still picture. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an apparatus, a method, and a program for performing processing according to the contents of an image.

  In Patent Document 1, an image captured by a camera capturing unit is temporarily stored in an image memory, and a captured state of a captured image in the image memory is analyzed by a CPU. And the like, and the arrangement balance of the subject, the degree of overlap, and the like are determined to prevent registration of an image having a high possibility of failed shooting. Further, even for an image that has been registered once, the registered image is read from the recording medium, the image content is judged, the registered image that has a high possibility of failed photographing is deleted, and the storage area of the recording medium is effectively used.

  In Patent Document 2, the CPU extracts a specific person portion from each captured image obtained by continuously capturing a plurality of (for example, six) images of the same person as a subject at the time of photographing, While analyzing this specific person part, the shooting state is evaluated for each “face orientation”, “face hiding condition”, and “eye opening condition”, and the evaluation result of the specific person part evaluated for each captured image Among them, a photographed image including a specific person portion that has been highly evaluated is identified.

  An image processing apparatus disclosed in Patent Literature 3 includes a registration device that registers a feature quantity of a physical part of an individual, an input unit that inputs image information, and a specifying unit that specifies a person who is a subject photographed in the input image. Detection means for detecting an image defect portion in the image of the subject, search means for searching for registration information of a specific person from the registration device, correction means for correcting the image defect portion based on the searched registration information, It is characterized by having.

  In Patent Document 4, since almost the same scene is captured before and after a frame image at a certain time, when the screen is completely stationary, if an average value of a plurality of frame images during the stationary time is taken, imaging input is performed. It is assumed that noise generated between the system and the recording system can be effectively removed.

  In Patent Document 5, an attribute inside a face area having an eye area determined to have eyes closed is changed, a video signal of the face area with the changed attribute is generated, and this is combined with a recording image. Display on LCD. Further, it is assumed that a “cut-out face area” of a desired image selected from the recording images is pasted on the face area where the blinking is detected.

Non-Patent Document 1 analyzes a complement target image from a huge amount of images and complements an unnecessary part of the image.
JP 2005-20196 A JP 2005-45600 A JP 2006-23949 A JP 2008-99327 A JP 2008-131094 A “Image Completion Technology in the Age of Web 2.0”, Internet URL [http://www.atmarkit.co.jp/news/200708/29/sc.html], search on August 28, 2008

  In Patent Document 1, the failed image can be deleted, but the failed image cannot be complemented. In Patent Document 2, a photographed image having a high evaluation can be specified, but a failed part cannot be complemented.

  In Patent Literature 3, since failure images are complemented by face information analysis, face information must be registered in advance, and cannot be applied to other than faces.

  In Patent Literature 4, noise can be removed, but the failed portion cannot be complemented.

  In Patent Document 5, there is no guarantee that such an image can be obtained with certainty unless an image necessary for the eye-medicating face image is secured in advance.

  In Non-Patent Document 1, a huge amount of image analysis is required, and the processing load is large.

  An object of the present invention is to easily and reliably secure an image that enables complementation of a failed image.

  An image processing apparatus according to the present invention includes a registration unit for registering a reference image or a feature amount thereof as a reference for success or failure of photographing, a desired image or a feature amount thereof, and a reference image registered by the registration unit or a feature thereof. A determination unit that determines whether or not a desired image is an image that has failed to be photographed according to the comparison result, and outputs a complementary image according to the determination result of the determination unit. A control unit that instructs the image output apparatus.

  Here, the image output device may include a device that does not include an imaging device such as an image server but accumulates images, in addition to a device that acquires an image such as an imaging device.

  The registration unit registers a failure reference image or its feature value that is a criterion for shooting failure, and the determination unit compares the desired image or its feature value with the failure reference image registered by the registration unit or its feature value. Then, according to the coincidence between the two, it is determined whether or not the desired image is an image for which photographing has failed.

  The registration unit registers the success reference image or its feature value, which is a reference for the success of shooting, and the judgment unit compares the desired image or its feature value with the success reference image registered by the registration unit or its feature value. Then, it is determined whether or not the desired image is an image that has failed to be photographed in accordance with the mismatch.

The registration unit registers a failure reference image or its feature value that is a reference for shooting failure,
The determination unit compares the desired image or the feature amount thereof with the failure reference image registered by the registration unit, and determines whether the desired image is an image that has been successfully photographed according to the mismatch between the images. .

  The registration unit registers a success reference image or its feature value that is a reference for the success of shooting, and the determination unit compares the desired image or its feature value with the success reference image registered by the registration unit, and In accordance with the coincidence, it is determined whether or not the desired image has been successfully captured.

  A success notifying unit for notifying that the acquisition condition of the success image has been prepared according to the determination result of the determining unit.

  A failure part specifying unit that specifies a failure part that matches the failure reference image registered by the registration unit or its feature amount from a desired image is provided.

  A failure notification unit is provided for notifying the determination result of the determination unit and / or the failure part specified by the failure part specifying unit.

  The failure portion specified by the failure portion specifying portion of the desired image is corrected by the complementary image input portion for inputting the complementary image output from the predetermined image output device and the complementary image input by the complementary image input portion. A correction unit.

  Of the failed parts identified by the failed part identifying unit, a selection unit that accepts selection of a failed part to be corrected is provided, and the correcting unit corrects the failed part that the selecting unit has accepted the selection with a complementary image.

  The selection unit accepts selection of a part to be restored in the image modified by the modification unit, and the modification unit returns the part accepted by the selection unit to the original failed part.

  An output unit that outputs an image corrected by the correction unit is provided.

  A first imaging unit that converts an electrical signal obtained by photoelectrically converting a subject image received through the imaging lens into an image and outputs the image; and the determination unit outputs an image output from the first imaging unit or its image Compare the feature amount with the reference image registered by the registration unit or the feature amount, and determine whether the image output by the first imaging unit is an image that has failed to be photographed according to the comparison result, The control unit instructs the second imaging device having the first imaging unit or the second imaging unit other than the first imaging unit to acquire a complementary image according to the determination result of the determination unit.

  The image output by the first imaging unit includes a through image output before the main imaging instruction is input to the first imaging unit.

  The complementary image includes still images or moving images acquired by the first imaging unit or the second imaging unit before and after the main imaging instruction is input to the first imaging unit.

  The feature amount of the reference image is at least one of the first feature amount related to the value of the image data and the second feature amount that is a feature amount other than the first feature amount related to the situation at the time of shooting the image data. including.

The image processing method according to the present invention includes a step in which a computer registers a reference image or a feature amount thereof as a reference for success or failure of photographing, a desired image or a feature amount thereof, and a registered reference image or a feature amount thereof. And determining whether or not a desired image is an image that has failed to be photographed according to the comparison result, and instructing a predetermined image output device to output a complementary image according to the determination result. And executing a step.
An image processing program for causing a computer to execute this image processing method is also included in the present invention.

  According to the present invention, a predetermined image acquisition device is instructed to acquire a complementary image in accordance with a comparison between a desired image or its feature value and a registered failed image or its feature value. In other words, if the possibility that a failed image is captured prior to the actual shooting is increased, a complementary image that complements the failed image can be secured, and the failed image can be reliably complemented later.

<First Embodiment>
FIG. 1 is a functional block diagram of a digital camera 10 according to a preferred embodiment of the present invention. First, in the digital camera 10 shown in the figure, an information processing apparatus (in addition to the CPU, peripheral devices necessary for information processing of the CPU, such as ROM, RAM, I / O, etc. are included. These are collectively referred to as a CPU for convenience). 112 controls each circuit in the digital camera 10 based on inputs from various buttons and keys of the operation unit 113 including a release switch, a mode dial, and the like. A program executed by the CPU 112 is stored in the ROM of the storage unit 119.

  The image pickup unit 102 receives an image pickup lens including a zoom lens and a focus lens, a motor for driving the image pickup lens, a motor driver for driving the motor in accordance with a command from the CPU 112, receives a subject image formed by the image pickup lens, performs photoelectric conversion, and performs analog conversion. It includes a solid-state imaging device such as a CCD or CMOS that outputs as an image signal.

  Now, when the still image shooting mode is set by the mode dial, the CPU 112 displays the frame images (through images) sequentially obtained from the imaging unit 102 on the display unit 110 so that the shooting angle of view can be confirmed.

  The imaging unit 102 includes an A / D converter, a gain adjustment circuit, an image signal processing circuit, a white balance correction circuit, a gamma correction circuit, a YC processing circuit, a luminance / color difference signal generation circuit, a sharpness correction circuit, a contrast correction circuit, and a captured image. An image processing unit including an outline processing unit that performs image processing including contour correction, a noise reduction processing unit that performs image noise reduction processing, and the like is processed, and an image signal is processed according to a command from the CPU 112 and converted into digital image data.

  The image data output from the imaging unit 102 is converted into a luminance signal (Y signal) and a color difference signal (Cr, Cl signal), and subjected to predetermined processing such as gamma correction, and then the VRAM in the storage unit 119. Stored in

  On the other hand, only the G pixel component is extracted from the image data by the high pass filter (HPF) 125. Then, the G pixel component extracted by the integration unit 126 is integrated and sent to the CPU 112. The CPU 112 calculates an average value of areas near the subject center (for example, four areas) arbitrarily set with the cross key among the image data of the divided areas (for example, 64 divisions) for one screen sent from the integrating unit 126. This is used as an autofocus (AF) evaluation value. The AF evaluation value is calculated every elapse of a predetermined period, and updated and stored in the storage unit 119 each time it is calculated. The CPU 112 determines the lens position at which the subject is focused according to the AF evaluation value. The determination method is arbitrary. For example, it is determined whether or not the AF evaluation value has substantially reached the vicinity of the maximum point. If the AF evaluation value has substantially reached the maximum point, the AF evaluation value is determined to be in focus. In this case, it is determined that the subject is not in focus (contrast AF). Alternatively, if the AF evaluation value exceeds a predetermined threshold, it can be determined that the subject is in focus.

  When the captured image is output to the display unit 110 on the monitor, the YC signal is read from the VRAM and sent to the video encoder 134. The video encoder 134 converts the input YC signal into a predetermined display signal (for example, an NTSC color composite video signal) and outputs the converted signal to the display unit 110.

  The YC signal of each frame processed at a predetermined frame rate is sequentially written in the VRAM, and a video signal generated from the YC signal is supplied to the display unit 110, so that the image being captured is displayed in real time. 110. The user can check the shooting angle of view from the video (through) displayed on the display unit 110.

  Here, when the release switch is half-pressed, the AE and AF processes are started. The AE / AF process is performed, and the recording operation is started when the release switch is fully pressed. The image data for recording acquired in response to the full press of the release switch is stored in the image storage unit 132. In the case of capturing a moving image, the above-described images are continuously captured, and each image constitutes a moving image as a frame image (frame image). Movie capture starts when the release switch is fully pressed for the first time and ends when the release switch is fully pressed for the second time. Alternatively, the video is captured only while the full press continues. The sound collected from the microphone 17 can also be captured synchronously with the acquisition of the moving image.

  The Y / C signal stored in the image storage unit 132 is compressed by the CPU 112 according to a predetermined format (for example, Exif (Exchangeable Image File Format) for a still image, MPEG4 for a moving image), and then stored as a file on an external recording medium. To be recorded. The image is recorded in the data part of the Exif file. The CPU 112 records shooting date / time information and the like in a predetermined tag (such as an image description tag) in the header portion of the Exif file. The recorded image can be transmitted to various electronic devices 200 such as a personal computer 200a, a printer 200b, and a PDA connected via the communication interface 76. The camera 10 is also connected to another camera 10 -B via the communication interface 76. The camera 10-B has the same configuration as the camera 10. In addition, it is assumed that the user has set an appropriate shooting angle of view so that the camera 10 and the camera 10-B capture a similar scene. Hereinafter, in order to distinguish between the camera 10 and the block of the camera 10-B, a subscript “-B” is attached to the block of the camera 10-B. When the sub imaging unit 20 is provided, it is assumed that the user sets an appropriate shooting angle of view so that the sub imaging unit 20 captures a scene similar to the imaging unit 102.

  The printer 200b prints a still image received from the camera 10 or the personal computer 200a on a mount made of a paper medium such as plain paper or photographic paper in a predetermined arrangement. Still images arranged in a predetermined arrangement can be output as electronic media such as “PDF” of Adobe Systems Inc. to a display, a recording medium, a personal computer connected via the Internet, and the like. In short, the output destination of the created image print is not limited to a specific one. In this specification, an electronic medium creation form on which a desired image can be arranged at a desired position is also called a mount.

  When the moving image shooting mode is set by the mode dial, the moving image recording operation starts in conjunction with the full pressing operation of the release switch, and the moving image recording operation stops when the release switch is pressed again. The recording operation may be performed while the release switch is continuously pressed, and the recording may be stopped by releasing the press. The moving image data is stored in the image storage unit 132 in the MPEG4 format, for example.

  When the playback mode is selected by the mode dial, the compressed data of the last image file (last recorded file) stored in the image storage unit is read out. The read image compressed data is decompressed into an uncompressed YC signal by the CPU 112 and added to the video encoder 134. The video encoder 134 creates an NTSC RGB color composite video signal from the input YC signal and outputs it to the display unit 110. Thereby, an image is displayed on the display unit 110.

  The display unit 110 corresponds to an external display device connected to an LCD, a finder, a video output terminal, or the like built in the camera 10. The video encoder 134 includes an OSD signal generation circuit 135. The OSD signal generation circuit 135 displays characters such as shutter speed, aperture value, number of images that can be shot, shooting date and time, warning instructions, etc. in accordance with instructions from the CPU 112, icons, and the like. Generate a signal to display the symbol. The signal output from the OSD signal generation circuit 135 is mixed with an image signal as necessary and supplied to the LCD 114. As a result, a composite image in which characters, icons, and the like are combined with the through image and the playback image is displayed.

  The image storage unit 132 is means for holding image data (still images, moving images) obtained by shooting, and for example, a memory card called smart media is used. The form of the recording medium is not limited to this, and it may be a PC card, a compact flash (registered trademark), a magnetic disk, an optical disk, a magneto-optical disk, a memory stick, etc., electronic, magnetic, optical, or a combination thereof A readable / writable medium can be used in accordance with the above-described method. A configuration in which a plurality of media can be mounted regardless of different types or the same type of recording media may be adopted. The means for storing the image file is not limited to a removable medium that can be attached to and detached from the camera body, but may be a recording medium (built-in memory) built in the camera.

  The CPU 112 is a control circuit that performs overall control of each circuit of the camera 10 and includes an operation unit including a release switch, a cross key, a power switch, a mode dial, an information position designation key, a strobe button (not shown), a zoom key, a menu / execution button, and the like. Based on the signal received from 113, the operation of the corresponding circuit is controlled, and display control in the display unit 110, strobe light emission control, AF auxiliary light emission control, autofocus (AF) control, automatic exposure (AE) control, etc. I do.

  Each time the power supply of the camera 10 is turned on by the power switch, power is supplied to each circuit of the digital camera 10 from a main power supply (not shown) composed of a battery mounted inside the camera body.

  The communication interface 76 can be connected to various information processing devices such as a personal computer and a printer, and transmits and receives data in accordance with commands from the CPU 112.

  The motion detection circuit 21 includes an acceleration sensor, a potentiometer, a gravity sensor, a tilt sensor, a posture sensor, an electronic compass, and the like. Specifically, the motion detection circuit 21 outputs the left-right motion information indicating the left-right motion direction and the amount of motion of the camera 100, the pressure detection unit for detecting the pressing force to the operation unit 113, particularly the release switch. A detection unit, a vertical movement detection unit that outputs vertical movement information indicating the vertical movement direction and movement amount of the camera 100, and a tilt movement information indicating the inclination direction and the inclination amount of the camera 100 with respect to the horizontal direction are output. A tilt motion detector is provided. The motion information includes the left and right motion information, the vertical motion information, the tilt motion information, the time when the detection of each motion is started, the time when the detection of each motion is completed, and the time when the detection of each motion is continued.

  Unlike the imaging unit 102 that is provided on the front surface of the housing of the camera 10 and is used for photographing a subject, the sub-imaging unit 20 is provided on the front surface of the housing and performs shooting by the photographer (hereinafter referred to as self-shooting). Do. The sub imaging unit 20 is a device that detects where on the screen of the display unit 110 the user is gazing at, and the detected information is input to the CPU 112. Detection of the line-of-sight direction of the eye to be examined is realized by projecting a light beam onto the anterior eye portion of the eye to be examined and detecting a reflection image by the cornea or the crystalline lens by a photoelectric conversion element. An apparatus for detecting the direction of the line of sight of an eye to be examined is well known.

  For example, the distance measuring unit 81 emits light from a light emitting unit to a subject within the shooting range of the imaging unit 102, receives light returned from the subject and returns the subject from an angle formed by the light emitting unit and the light receiving unit. A triangulation distance sensor that calculates the distance up to can be used. Alternatively, various known techniques can be adopted as long as the object is a means for measuring a distance such as a sensor applying the TOF (Time of Flight) principle or a distance image (3D image) of a compound eye camera.

  A touch panel 114 is stacked on the display unit 110. When the user presses a corresponding part of the display unit 110 with a finger or a pen, information indicating the pressed position is output to the CPU 112.

  A speaker 85 is connected to the CPU 112 via an audio processing circuit 84. The audio processing circuit 84 drives the speaker 85 based on the notification sound data (shutter sound, warning sound, etc.) sent from the CPU 112, thereby outputting the notification sound and the sound recorded together with the moving image.

  In the video shooting mode, the analog audio signal acquired from the microphone 17 is converted into digital data after being amplified by the audio processing circuit 84 and is subjected to data processing such as being compressed in a predetermined compression recording format (such as MPEG4). The image is stored in the image storage unit 132 together with the acquired image. Even in the still image shooting mode, it is possible to record audio before and after the main shooting instruction. In addition to the sound stored in the image storage unit 132, various notification sound data such as operation sound, shutter sound, and AF in-focus sound stored in advance in the storage unit 119 are decoded and analog sound by the sound processing circuit 84. After data processing such as signal conversion and amplification is performed, the data is output to the speaker 85. The speaker 85 outputs sound according to the analog sound signal output from the sound processing circuit 84.

  An RTC (real time clock) 15 outputs current date and time information. This current time information can be recorded together with the image as shooting date / time information of the recording image by the imaging unit 102.

  The voice recognition circuit 54 performs voice recognition on the voice directly acquired from the microphone 17 or the voice represented by the voice data designated by the CPU 112 out of the voice data stored in the image storage unit 132, and the recognition result is converted into text. Performs speech recognition processing that is output as data. A speech database 74 is connected to the speech recognition circuit 54, and the speech database 74 stores pattern data referred to for pattern matching during speech recognition processing. The voice recognition circuit 54 performs voice recognition with reference to the pattern data stored in the voice database 74. In the voice database 74, for example, at the time of various assumed shootings, such as crowded voices collected at public facilities such as a station where an unspecified number of people come and go, and noises of car traffic on the sidewalk along the road An audio pattern indicating the situation is registered.

  The GPS receiver 120 obtains current position information of the camera 10 by receiving radio waves from GPS satellites and measuring the distance from the orbit. This current position information can be stored in the image storage unit 132 together with an image (for example, as meta information or header information) as shooting position information of an image for recording by the imaging unit 102.

  A microphone 17 is installed on the rear surface of the camera casing, and these various devices are provided around the display unit 110. Moreover, although the sub imaging part 20 can be provided as a camera housing front surface or an expansion device of the camera 10, it is not necessary to provide it.

  The CPU 112 acquires user information that is the basis of image processing. For example, the user information includes position information from the GPS receiver 120, motion information indicating the motion detected by the motion detection circuit 21, shooting date / time information from the RTC 15, operation content information on the operation unit 113 and the touch panel 114, and composition from the CPU 112. Information, subject information, image quality information, recognized voice information from the voice recognition circuit 54, or an image file in which these are recorded as meta information or header information.

  The image analysis unit 145 analyzes the image in the image storage unit 132, and creates and outputs composition information, subject information, and image quality information according to the result. The composition information includes the presence / absence of a person and the position of a main subject (typically a focused subject, but may be a subject with face detection). The subject information includes the position of the face area, subject size, person name, presence / absence of subject motion, direction of motion, and speed of motion. Note that various known techniques may be used for composition analysis, subject extraction, face detection, and image quality analysis (for example, Japanese Patent Application Laid-Open Nos. 2007-97090 and 2005-354606 by the present applicant). Such information may be recorded as supplementary information of the image.

  The image quality information includes image information indicating the presence / absence of blurring / blurring and the presence / absence of eye-meditation. As for shake detection, as in the case of conventional camera shake detection, it is determined that there is camera shake when the size of the acceleration vector detected by the motion detection circuit 21 exceeds a predetermined threshold. The blur detection can be performed based on, for example, edge detection (for example, Japanese Patent Application Laid-Open No. 2005-122721 by the present applicant). A known technique can also be used for determining whether or not eyes are closed (for example, Japanese Patent Application Laid-Open No. 2007-49631 by the present applicant).

  The operation content information includes the content of the operation input between the shooting mode setting and the half-press of the shutter button. The operation content information is stored in the storage unit 119 by the CPU 112 every time there is an operation input to the operation unit 113 or the touch panel 114. Such information may be recorded as supplementary information of the image.

  The unsuccessful image registration unit 141 operates to select an image to be registered as a unsuccessful image among desired images such as an image stored in the image storage unit 132, an image input from the communication I / F 76, and an image read from a portable storage medium. It accepts via the part 113. Then, information indicating the classification of the failed image is associated with the image designated from the operation unit 113 and registered in the image storage unit 132. Alternatively, selection of an image to be registered as a successful image among desired images secured from the image storage unit 132 or the like is received via the operation unit 113. Then, information indicating the classification of the successful image is associated with the image designated from the operation unit 113 and registered in the image storage unit 132. Alternatively, based on the information acquired by the image analysis unit 145, the failure image and the success image are classified from the images acquired by the imaging unit 102, the sub imaging unit 20, and the camera 10-B, and information indicating the classification result is displayed. The image storage unit 132 can also be registered in association with each image.

  Which image is registered as a failed image or a successful image depends on an instruction input to the operation unit 113 by the user, and any image may be registered. For example, in a desired image, the user arbitrarily selects an image with a human subject facing sideways or rearward or with eyes closed, etc., via the operation unit 113, and designates it as a failed image. Alternatively, the user selects an image with a human subject facing the front side via the operation unit 113 and sets it as a successful image.

  Instead of registering the failure image or the success image itself, designation of an area in the image that the user considers to be failure or success may be received from the operation unit 113, and the portion may be registered as the failure image or the success image. Further, if a unique feature amount indicating the portion can be extracted by the image analysis unit 145, the feature amount may be registered as a substitute for the failed image or the successful image. The feature amount includes the length of the exposed black eye in the vertical direction (unit: pixels), the number of eyes detected from the face area (2, 1, or 0), and the position of the moving subject other than the human subject. There is information. Such a feature amount may be created by image analysis of the image analysis unit 145. If moving images before and after still image recording are recorded by the camera 10-B and the sub imaging unit 20, the moving subject portion in the still image can be specified from the difference between the moving image before and after the still image shooting and the still image. , That part can be a failed image. It is also possible to create a motion vector from the difference and use it as a primary feature value.

  The type of sound recognized at the time of image acquisition and the presence or absence of noise can also be registered as one of the feature amounts of the failed image or the successful image. An image taken when noise is detected is likely to cause an unnecessary subject such as a car or a group of people to pass behind or in front of the main subject, and the noise level (for example, 70 decibels) is set as the failure image. It can be registered as a feature value. Or, in the case of a group photo, since shooting will usually start after all people are quiet, the noise level (for example, 50 dB) at which the user took a group photo designated as a successful image via the operation unit 113 Are registered as feature quantities. In addition, the number of human subjects detected by the image analysis unit 145 by known face detection or edge detection may be registered as a feature amount.

  Alternatively, the instantaneous shake amount (acceleration) of the camera of the motion detection circuit 21 at the time of shooting can also be registered as a feature amount. For example, camera shake has occurred, and the shake amount at the time of shooting an image selected as a failed image can be registered as a feature amount of the failed image. Alternatively, the EV value and color temperature of a failed image taken with backlighting under a clear sky can be registered as the feature amount of the failed image. Further, the edge feature amount (edge sharpness) of the out-of-focus image selected as the failure image can be registered as the feature amount of the failure image. Such a feature amount can be registered as a failure image combined with coordinate information of a designated eye-meditation region and its feature amount.

  Alternatively, the position and size of the main subject in the image selected as the failed image because the main subject is too small, or the distance to the main subject or the other subject measured by the distance measuring unit 81 is also a feature of the failed image. Can be registered as a quantity.

  Alternatively, if an image that has been blacked out due to underexposure (underexposure) is selected as a failed image, or an image that is overexposed and overexposed is selected as a failed image, the EV value is set. It can be registered as the feature amount of the failed image.

  Alternatively, it is possible to register the luminance information of the failed image in which the highlight portion and the shadow portion of the luminance range of the image do not fall within a predetermined latitude and the highlight portion skips or the shadow portion is collapsed as the feature amount of the failed image. .

  In other words, the feature amount of the failed image or the successful image is information related to the value of the image data and an index or numerical value derived from the related information (whether there is a human subject, the size of the human subject, whether the human subject is meditating, The number of the subject's eyes and the orientation of the subject's face, luminance information, histogram of the luminance value of each pixel included in the face area, presence / absence of a moving subject, etc. (hereinafter referred to as a primary feature) In addition, the information is not derived from the value of the image data itself, but various indexes and numerical values (EV value, focal length, ambient noise level, distance to the subject, etc.) relating to the situation at the time of shooting the image data. (Hereinafter referred to as “secondary feature amount”), and it is up to the user to register which of them is associated with the failure image or the success image. Hereinafter, the registered failure image, success image, and various feature quantities associated with them are referred to as failure / success condition information.

  The failure image detection unit 142 continuously outputs an arbitrary image (not only a recorded image but also the imaging unit 102) based on the failure / success condition information registered in the image storage unit 132 by the failure image registration unit 141. The presence or absence of the failure portion (including the through image) and the location thereof are specified, and the CPU 112 is notified. Based on the notification of the failed part, the CPU 112 displays a warning or a message or outputs a voice, or replaces the failed part with a successful part of the success image. This process will be described later.

  The digital camera 10 may be any device that can acquire a subject image as image data, and may include a mobile phone with a camera, a personal computer with a camera, a digital video camera, a digital still camera, and the like.

  The storage unit 132 stores character information and template data to be combined with image data, and the image processing unit 144 adds desired data to a desired image in accordance with an instruction from the CPU 112 or a selection operation from the operation unit 113. Is synthesized.

  FIG. 2 is a flowchart showing an outline of the imaging support process. This process starts according to the setting of the shooting mode. Alternatively, it may be started in response to an instruction for photographing preparation operation (half-pressing the shutter button) being input to the operation unit 113. A program that causes the CPU 112 to execute this processing is stored in the storage unit 119. The “CPU 112” is not built in the camera 10 itself, but may include another electronic device, for example, a built-in computer 200a. That is, the execution subject of this process is the CPU of the personal computer 200a, and the control target can be an imaging device separate from the personal computer 200a.

  Further, in this specification, “matching” includes substantially matching, that is, not only completely matching but also including the case where the error falls within a predetermined allowable range even if not matching completely. .

  In S <b> 1, acquisition of a through image is started, analysis of the content and size of sound collected by the microphone 17 and sound collected by the speech recognition circuit 54, luminance information of the image and motion in the image by the image analysis unit 145 are performed. Detection of a certain object, detection of an eye-meditation area, detection of the diameter of an exposed black eye, detection of the number of eyes, measurement of a distance to a subject by a distance measuring unit 81, and detection of motion of the camera 10 by a motion detection circuit 21 Control each block to Information obtained from each block by these controls is generically referred to as shooting state information.

  In S2, the feature amount associated with the failed image included in the failure / success condition information is compared with the shooting situation information, and whether the shooting situation information includes information that matches the feature amount of the failed image. Judge whether or not. If the shooting situation information includes information that matches the feature amount of the failed image, the process proceeds to S3. Otherwise, the process proceeds to S4.

  Alternatively, the feature amount associated with the success image included in the failure / success condition information is compared with the shooting state information, and the shooting state information includes information that matches the feature amount of the success image or the success image. Judge whether or not. If the shooting situation information does not include information that matches the feature amount of the successful image, the process proceeds to S3. Otherwise, the process proceeds to S4.

  Note that S2 may be performed by the failed image detection unit 142.

  In S3, the part on the through image that matches the image part registered as the failed image is specified. This is because, for example, the partial area of the through image included in the set area and the failed image are set in an arbitrary position so that the same area as the size of the failed image scans the entire through image. If a difference image is created and the difference image becomes 0 (no color data), it is determined that the registered part of the failed image is in the through image.

  Then, the image indicating the frame failure, coloring, color reversal, etc. for the failed image part in the specified through image, the warning message that there is a failure part, and shooting at the set angle of view A warning that there is a possibility of being obtained is displayed on the display unit 110 by an icon, a message, or other images, or is reproduced from the sound processing circuit 84 and the speaker 85 by a chime sound, a warning message, or other sounds. In addition, the OSD circuit 135 may be instructed to generate a video indicating the determination result of S3, or the audio processing circuit 84 may be instructed to reproduce such a warning message. Alternatively, when the shutter button is fully pressed and a shooting instruction is input from the operation unit 113 despite the detection of such a failed part, the CPU 112 locks the shooting by invalidating or ignoring the shooting instruction. However, it is also possible to reproduce a message indicating that shooting was not performed and that the shooting instruction was not followed. Although illustration is omitted, when shooting is locked, the process may not return to S5 but return to S1.

  Even if there is no portion corresponding to the failed image in the through image, if it is determined that the feature amount at the time of acquiring the through image matches the feature amount of the failed image, for example, the surrounding sound collected at the time of acquiring the through image The EV value at the time of acquisition of the through image calculated by the CPU 112 matches the EV value of the failed image of backlight, overexposure, or underexposure, or when the through image acquired by the motion detection circuit 21 is acquired. The acceleration applied to the camera 10 exceeds the acceleration corresponding to the camera shake failure image, or the image analysis unit 145 has a moving object from the through image, the human subject who has meditated eyes, the face not facing the front (both eyes are not detected) If a face is detected, such a warning is displayed or aloud.

  In addition, the type of failure may be notified. For example, if it matches the feature value of eye meditation, a message such as “There is eye meditation” will be displayed and loud, and if the noise is loud, a message such as “Noise that is currently disturbing shooting has been observed” will be displayed and loud. Output. Alternatively, if an underexposed feature is detected, a warning to that effect is output.

  Since the through image is an image that is periodically acquired, the feature amount at the time of acquiring the through image is also periodically acquired. The period may not be the same as the acquisition period of the through image.

  Further, if it is determined that the registered part of the failed image is in the through image, or if it is determined that the feature amount at the time of acquiring the through image matches the feature amount of the failed image, the camera 10-B (or the sub imaging unit) 20. The same applies to the following, instructing the start of acquisition of the complementary image. Upon receiving this instruction via the communication I / F 76, the camera 10-B subsequently acquires a periodic image by the imaging unit 102-B, and sequentially uses the acquired image as a complementary image to the image storage unit 132. Store in -B. This may be acquisition of a frame image of a moving image or continuous shooting of still images. You may acquire the image for a complement by the continuous shooting of the still image by the imaging part 102 of the camera 10 itself instead of the camera 10-B, and moving image acquisition. Furthermore, both the camera 10 and the camera 10-B may acquire a complementary image. Furthermore, the desired personal computer 200a connected to the communication interface 76 may be requested to output a complementary image, and the personal computer 200a may output the complementary image to the camera 10 in response to the request. As described above, when the possibility of shooting the failed image increases, a complementary image for correcting the later failed image is secured.

  However, in order to prevent this complementary image from being stored indefinitely, it is preferable to discard old images (frame images) so that the number of complementary images is within a predetermined number or a predetermined time length. For example, if the threshold value for starting discard is set in the storage unit 119 as “5”, when six or more complementary images are obtained, the oldest image is deleted in order, and the number of images is deleted. Deletion is stopped when the threshold is reached. Alternatively, if the time threshold for starting discarding is set in the storage unit 119 as “10 seconds”, a moving image frame image older than the moving image frame image that is 10 seconds back from the current time is deleted.

  In S4, the OSD circuit 135 is instructed to synthesize a video signal of a notification message indicating that a condition suitable for shooting is satisfied. Alternatively, the audio processing circuit 84 is instructed to reproduce the notification message. In addition, an instruction to stop acquisition of the complementary image may be output to the camera 10-B.

  In S5, it is determined whether or not an instruction to start photographing has been input to the operation unit 113 (whether or not the shutter button has been fully pressed). In the case of Yes, it progresses to S6, and in No, it returns to S1.

  In S6, a shooting operation is performed in response to a shooting start instruction input. As a result, the recording image acquisition operation by the imaging unit 102 is performed, and the image is stored in the image storage unit 132. Further, the feature amount at the time of inputting an instruction to start shooting is stored in the image storage unit 132 in association with the image.

  Further, after storing the image, the camera 10-B is instructed to end acquisition of the complementary image for a certain time (for example, after 5 seconds). When the camera 10 -B receives this instruction via the communication I / F 76, the camera 10 -B stops acquiring images.

  Note that even if a failed image is detected in S <b> 2, for subsequent through images, the CPU 112 determines whether there is a portion on the through image that matches the registered image portion of the successful image, and the feature amount of the successful image. If there is a match between the through image and the success image part, or if there is a feature amount and a success image feature amount when the through image is acquired, the shooting conditions are in place. A notification message to that effect may be output. If the shooting is locked, it is canceled and the main shooting instruction is accepted. This further increases the possibility of successful image recording. In addition, if there is a main shooting instruction while the shooting lock is released, and imaging and image storage are performed, the complementary image acquired at the same timing as when the image is acquired may be discarded. . This is because the same complementary image is not necessary as long as the success image is secured.

  Note that a video indicating the presence or absence of a portion on the through image that matches the image portion of the registered successful image may be similarly indicated to the OSD circuit 135.

  As described above, when the failure image registered in advance and the feature amount thereof are obtained at the time of acquiring the through image, the possibility of the user taking the failure image can be reduced by outputting a warning. In addition, when a pre-registered success image and its feature amount are obtained when a through image is acquired, a notification that the shooting conditions are ready can be output to further increase the possibility that the user will take a success image. it can.

<Second Embodiment>
FIG. 3 is a flowchart of image complement processing according to the second embodiment. A program that causes the CPU 112 to execute this processing is stored in the storage unit 119. However, this program can also be executed by an information processing apparatus other than the camera 10, for example, the personal computer 200a. This personal computer has the same configuration as the CPU 12, the communication I / F 76, the storage unit 119, the image storage unit 132, the operation unit 113, the failed image registration unit 141, the image analysis unit 145, the display unit 110, the video encoder 134, and the OSD circuit 135. It shall be equipped with.

  In S <b> 11, a captured image (either a still image or a moving image frame image) or a registered failed image is input from the image storage unit 132. If there are a plurality of these images, enter each one. When this processing is performed by the personal computer 200a, the photographed image is input from the camera 10 via the communication I / F 76 or the portable storage medium, and the camera 10- is connected via the communication I / F 76 or the portable storage medium. A complementary image is input from B (or camera 10).

  In S12, the same processing as in S2 is performed on the captured image instead of the through image. If there is a region in the photographed image where the failed image matches, or if the feature amount corresponding to the photographed image matches the feature amount of the failed image, the process proceeds to S13. Otherwise, the process ends. To do. This process will be described in detail later.

  In S13, an image showing frame surrounding, coloring, color reversal, and the like for a region (failed portion) in the captured image that matches the failed image, a warning message that there is a failed portion, and other images showing the determination result in S12. The generation is instructed to the OSD circuit 135. Alternatively, the audio processing circuit 84 may be instructed to reproduce a predetermined warning sound together with the display of the warning message. Note that the content of the message in S3 is a warning that suggests that a photographer who is going to take a photograph may take a failed image, but the message in this step S13 is a failed part of the image that has already been taken. This is a warning that suggests that there may be.

  In S <b> 14, a desired portion to be corrected among the failed portions is selected via the operation unit 113. The process proceeds to S15 according to the selection.

  In S15, the selected correction portion is replaced with the corresponding portion of the complementary image stored in the camera 10-B that has the feature amount of the successful image. Then, the corrected image after replacement is displayed on the display unit 110.

  FIG. 4 is a flowchart showing the details of the processing (S12 and S13) for determining whether or not the photographed image or its feature amount matches the failed image or its feature amount, and notifying if it matches. . The following flowchart is a determination for a part of the primary feature quantity or the secondary feature quantity described in the first embodiment, but the same determination can be made for other feature quantities.

  In S <b> 21, a captured image that may fail may be extracted from the image storage unit 132. This is because, for example, it is determined whether the captured image or its feature value matches the success image or its feature value, and the captured image that does not match the success image or its feature value at all has a possibility of failure. Extract as a completed image. The extracted captured images with the possibility of failure are displayed in a list on the display unit 110, and an image to be processed from S22 onward is selected from the operation unit 113, and the acquired image is acquired again. Other images may be excluded from captured images that may fail.

  In S22, the primary feature amounts corresponding to the captured image and the failed image acquired in S21 are compared, and it is determined whether or not they match. If yes, go to S23, if no, go to S24.

  In S23, the area of the captured image having the primary feature amount of the failed image is specified, and the OSD circuit 135 is instructed to generate a failed part instruction video indicating the portion.

  In S24, it is determined whether or not the subject exists at a distance closer than the distance of the focused subject. This is because the focal length is converted into the distance to the in-focus subject, the distance to the in-focus subject, and the distance to the subject measured by the distance measuring unit 81 (including the in-focus subject and other subjects) Are compared to determine whether or not there is a subject (front side subject) that is closer than the distance to the focused subject. If Yes, the process proceeds to S25, and if No, the process proceeds to S26. Before making this determination, the distance measuring unit 81 performs distance calculation only for an area in which a person detected by face detection or an object (such as a car) other than a person detected by pattern matching exists. The distance corresponding to each area is stored in the image storage unit 132, and the distance data of each subject is stored.

  In S25, the presence portion of the near-side subject in the captured image, that is, the presence area of the near-side subject is specified, and the OSD circuit 135 is instructed to generate a failed portion instruction video indicating the portion.

  In S26, the size of the focused subject in the captured image is compared with the subject size included in the feature amount of the failed image, and it is determined whether the former is equal to or less than the latter. In the case of Yes, it progresses to S27. In the case of No, this process is finished.

  In S27, the OSD circuit 135 is instructed to generate a failed part instruction video indicating the presence part of the focused subject.

  FIG. 5 shows an example of the failed image portion B and the successful image portion G. Here, the failed image portion B is a meditated eye in the subject's face region and its vicinity region, and the successful image portion G is the eye in the subject's face region with its eyes fully open and its vicinity region. is there. These can be stored in the image storage unit 132 as primary feature amounts.

  FIG. 6 illustrates a first example of an image (through image or captured image) acquired from the imaging unit 102, the sub imaging unit 20, or the like. The subjects SB1 and SB2 in the image have their eyes closed and coincide with the failed image portion B in FIG. In addition, the subject SB3 has completely opened eyes and coincides with the successful image portion G.

  Therefore, the image of FIG. 6 is a through image, and when the image capturing support process of FIG. 2 is performed on this image, a video or audio warning indicating the failed image portion B of the subjects SB1 and SB2 is output. Alternatively, the image in FIG. 6 is a captured image, and when the image complementing process in FIG. 3 is performed on this image, a video or audio warning indicating the failed image portion B of the subjects SB1 and SB2 is output.

  FIG. 7 shows a second example of an image (through image or captured image) acquired from the imaging unit 102 or the like. Subjects SB4 and SB5 exist in the image. The subject SB4 is a focused subject, but the subject SB5 is a movement of the subject SB2 by comparing the captured image obtained by the image analysis unit 145 with the complemented images taken by the camera 10-B or the sub-imaging unit 20. (However, it is assumed that the motion detection circuit 21 has not detected motion and the camera 10 is stationary). The subject SB2 is a moving object such as a car that suddenly jumps out at the time of shooting. In this case, a video indicating that the subject SB5 is a failed image portion, here, a video output of a broken line T1 surrounding the subject SB5 is performed.

  Alternatively, even when the subject SB5 is a focused subject and the subject SB6 is closer than the focused subject SB5, video output of a broken line T1 surrounding the subject SB5 is performed.

  FIG. 8 shows a third example of an image (through image or captured image) acquired from the imaging unit 102. In the image, there are subjects SB6 and SB7 that are detected to be persons. The human subject SB6 is a focused subject, but the human subject SB7 is not focused, and the subject size of the SB7 is smaller than the focused subject SB6. In this case, a video indicating that the subject SB7 is a failed image portion, here, a video output of a broken line T2 surrounding the subject SB7 is performed.

  FIG. 9 shows an example of the correction of the failed image part (S14). The image I2 is a captured image stored by the camera 10 in S6, and the images I1 and I3 are complementary images stored by the camera 10-B or the camera 10 before and after the image I2 is captured in S3. The images I1 to I3 include subjects SB11, SB12, and SB13. The subjects SB11 and SB12 of the image I2 have their eyes closed, and these eye regions are failed image portions. In S13, it is displayed that the respective eye areas of the subjects SB11 and SB12 of the image I2 are failed image portions P2-1 and P2-2. In S14, P2-1 is selected as a failed image portion to be corrected. , P2-2 are both selected.

  Note that if the failed image registration unit 141 deletes the unselected failure image portion and its feature amount from the failure / success condition information, only the failed image portion and the feature amount that the user actually thinks to have failed. Remains in the failure / success condition information, and the failure image portion and its feature amount with high accuracy are gradually accumulated as failure / success condition information.

  In S15, CPU 112 identifies image portions (complementation candidates) corresponding to selected failed image portions P2-1 and P2-2 from the complementing images. This is because the subject of the complementary image I1 or I3 existing in the same position as the subject SB11 and SB12 of the image I2, here the eye region of the subject SB11 and SB12 is detected by a known eye region detection technique (for example, Patent Document 5). The identified eye area becomes a candidate for complementation.

  Next, the CPU 112 compares the feature amount of the supplement candidate with the feature amount of the successful image registered in advance, and determines whether or not they match. If the two match, the complement candidate is used as a complement image portion. If the two do not match, the replacement candidate is excluded.

  Complementary candidates corresponding to the failed image portion P2-1 of the subject SB11 of the image I2 in FIG. 9 are P1-1 of the image I1 and P3-1 of the image I3. Further, the complement candidates corresponding to the failed image portion P2-2 of the subject SB12 of the image I2 are P1-2 of the image I1 and P3-2 of the image I3.

  P1-1 of the image I1 and P3-2 of the image I3, which are complement candidates, are eye regions where eye meditation has occurred, and the feature amounts thereof do not match the feature amounts of the successful image. Therefore, it is excluded from the complement candidates.

  P1-2 of the image I1 and P3-1 of the image I3, which are complement candidates, are eye regions that are completely open, and their feature amounts match the feature amounts of the successful image. Therefore, it is set as a complementary image portion.

  The CPU 112 replaces the failed image part with the corresponding complementary image part. For example, P2-1 is replaced with P3-1 and P2-2 is replaced with P1-2. Thereby, the corrected image I ′ in which the failed image portion is replaced with the corresponding complementary image portion is completed. The completed corrected image I ′ is displayed on the display unit 110. If the failed image part is a very small part, it is less uncomfortable to replace the peripheral area including that part. For example, when replacing small parts in the face area such as eyes, nose, and mouth, the face area including those parts may be replaced. If only one of P2-1 and P2-2 is selected as the failed image portion to be corrected, only the selected one is replaced.

  When there are a plurality of complementary image portions corresponding to the same failed image portion, a plurality of images in which the failed image portions are replaced by the respective complementary image portions are created, and a desired correction image is selected from the images via the operation unit 113. Alternatively, only the selected corrected image may be stored in the image storage unit 132.

  For example, if the image I4 in FIG. 9 is secured as a complementary image and the complementary image portion P4-1 corresponds to the failed image portion P2-1, the failed image portion P2-1 is replaced with the complementary image portion P4. A corrected image I′2 replaced with −1 is created, and a desired one of the corrected image I ′ and the corrected image I′2 is selected, and only the selected one is stored. When selecting a corrected image, the degree of smile is determined based on the image that is most likely to be selected, for example, the well-known facial expression recognition technique, the degree of bending of the mouth, the degree of rising corners, the size of the eyes, the degree of wrinkles, etc. It may be quantified and displayed by switching sequentially from the higher to the lower. Note that it is not necessary to select one corrected image to be stored, and any desired number of corrected images can be selected and left.

  FIG. 10 shows another example of the correction of the failed image part (S14). An image J2 in FIG. 10 is a captured image that has a stationary subject SBs in focus but has a moving subject SBm as a failed image portion Qf. Assume that the images J1 and J3 are one frame of a still image or a moving image captured by the camera 10-B before and after the image J2 is captured. In this case as well, as described above, the CPU 112 selects, as the complementary image portion Qs, the one that matches the feature amount of the successful image from among the complementary candidates corresponding to the failed image portion Qf, and selects the failed portion Qf as the complementary image portion Qs. Replacement and corrected image J3 is obtained. It should be noted that it is preferable that the still image J2 and the moving image frame J1 have the same image quality, because this replacement does not cause any inconsistency with surrounding image quality.

  The created corrected image can be stored in the hard disk of the personal computer 200a or printed out by the printer 200b. Of course, the user is not forced to use the corrected image, but the CPU 112 causes the operation unit 113 to select a desired portion of the generated corrected image, and only the selected portion is the original failed image before correction. After returning to the part, the image may be output to the image storage unit 132 or the display unit 110, or a failed image that is not corrected at all may be output to the image storage unit 132 or the display unit 110. To return to the original failed image portion, the images before and after correction are temporarily stored in the storage unit 119, and if an instruction to return is input, the corrected image is discarded from the storage unit 119. Then, the image before correction may be held in the storage unit 119 as it is. Thereafter, the held image can be output to a desired output destination such as the image storage unit 132, the display unit 110, the printer 200b, or a recording medium.

<Other embodiments>
In S2 of the imaging support process of FIG. 2 of the first embodiment, the process may proceed to S3 if it is determined that the feature values of both the through image and the success image are inconsistent, and to S4 if it is determined to match. Alternatively, the process may proceed to S3 if it is determined that the feature values of both the through image and the failed image match, and to S4 if it is determined that the feature values of both the through image and the success image match. Alternatively, the process may proceed to S3 if it is determined that the feature values of both the live view image and the success image do not match, and to S4 if it is determined that the feature values of both the live view image and the failure image do not match. The feature values of the live view and the success image do not match, and the feature values of the live view and the failure image do not match, or the live view and the success image match, and the live view and the fail image match. If the feature amounts of both images match, the determination may be impossible and the process may proceed from S2 to S6. At this time, it may be displayed that the determination is impossible. In S12 of the image complementing process of FIG. 3 of the second embodiment, the same determination and branching may be performed for the captured image.

Functional block diagram of digital camera Flow chart of shooting support processing Flow chart of image complement processing A flowchart of processing for determining whether or not a photographed image or its feature value matches a failed image or its feature value, and notifying if it matches. The figure which shows an example of a failure image part and a success image part The figure which shows the 1st example of the acquired image The figure which shows the 2nd example of the acquired image The figure which shows the 3rd example of the acquired image The figure which shows an example of correction of the failure image part The figure which shows another example of correction of the failure image part

Explanation of symbols

10: Camera, 112: CPU, 141: Failure image registration unit, 142: Failure image detection unit, 200: Electronic device

Claims (18)

A registration unit for registering a reference image or its feature value as a reference for success or failure of shooting;
The desired image or its feature value is compared with the reference image or its feature value registered by the registration unit, and it is determined whether the desired image is an image that has failed to be photographed according to the comparison result. A determination unit to
A control unit that instructs a predetermined image output device to output a complementary image according to a determination result of the determination unit;
An image processing apparatus comprising: The registration unit registers a failure reference image or a feature amount thereof serving as a reference for shooting failure,
The determination unit compares the desired image or the feature amount thereof with the failure reference image registered by the registration unit or the feature amount thereof, and determines whether the desired image has failed to be photographed according to a match between the images. The image processing apparatus according to claim 1, wherein it is determined whether or not there is. The registration unit registers a success reference image that is a reference for success of shooting or a feature amount thereof,
The determination unit compares the desired image or the feature amount thereof with the success reference image registered by the registration unit or the feature amount thereof, and determines whether the desired image has failed to be photographed according to the mismatch between the images. The image processing apparatus according to claim 1, wherein it is determined whether or not there is. The registration unit registers a failure reference image or a feature amount thereof serving as a reference for shooting failure,
The determination unit compares the desired image or a feature amount thereof with a failure reference image registered by the registration unit, and determines whether the desired image is a photographed image according to a mismatch between the two. The image processing apparatus according to claim 2, wherein the image processing apparatus determines whether or not. The registration unit registers a success reference image that is a reference for success of shooting or a feature amount thereof,
The determination unit compares the desired image or a feature amount thereof with the success reference image registered by the registration unit, and whether the desired image is an image that has been successfully photographed or not according to a match between the two. The image processing apparatus according to claim 2, wherein the image processing apparatus determines whether or not.   The image processing apparatus according to claim 4, further comprising a success notification unit that notifies that a condition for acquiring a success image has been established according to a determination result of the determination unit.   5. The image processing apparatus according to claim 2, further comprising a failure part specifying unit that specifies a failure part that matches a failure reference image registered by the registration unit or a feature amount thereof from the desired image.   The image processing apparatus according to claim 7, further comprising a failure notification unit that notifies a determination result of the determination unit and / or a failure part specified by the failure part specifying unit. A complementary image input unit for inputting a complementary image output by the predetermined image output device;
A correction unit that corrects the failed part specified by the failed part specifying part in the desired image by the complementary image input by the complementary image input unit;
An image processing apparatus according to claim 7 or 8. Among the failed parts identified by the failed part identifying unit, comprising a selection unit that accepts selection of a failed part to be corrected,
The image processing apparatus according to claim 9, wherein the correction unit corrects a failure portion for which the selection unit has received a selection with the complementary image. The selection unit accepts selection of a portion to be returned to the original image corrected by the correction unit,
The image processing apparatus according to claim 9, wherein the correction unit returns the portion that the selection unit has received the selection to the original failure portion.   The image processing apparatus according to claim 9, further comprising an output unit that outputs an image corrected by the correction unit. A first imaging unit that converts an electrical signal obtained by photoelectrically converting a subject image received through an imaging lens into an image and outputs the image;
The determination unit compares the image output from the first imaging unit or the feature amount thereof with the reference image registered by the registration unit or the feature amount thereof, and determines the first imaging according to the comparison result. Determine whether the image output by the copy is an image that failed to be captured,
The control unit instructs the second imaging device having the first imaging unit or a second imaging unit other than the first imaging unit to acquire the complementary image according to the determination result of the determination unit. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 13, wherein the image output from the first imaging unit includes a through image output before a main imaging instruction is input to the first imaging unit.   The image according to claim 13, wherein the complementary image includes a still image or a moving image acquired by the first imaging unit or the second imaging unit before and after a main imaging instruction is input to the first imaging unit. Processing equipment.   The feature amount of the reference image is a second feature amount that is a feature amount other than the first feature amount related to the value of image data and the first feature amount related to the situation at the time of shooting the image data. The image processing apparatus according to claim 1, comprising at least one of them. Computer
A step of registering a reference image or its feature value as a reference for success or failure of shooting;
Comparing a desired image or its feature quantity with the registered reference image or its feature quantity, and determining whether or not the desired image is an image that has failed to be photographed according to the comparison result; ,
Instructing a predetermined image output device to output a complementary image according to the determination result;
An image processing method for executing.   An image processing program for causing a computer to execute the image processing method according to claim 17.

Images