JP5488961B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  Conventionally, in a digital camera, a human face is detected from a captured image, and the detected face area is focused so as to be within the depth of field (face priority AF), or the face area is in an appropriate exposure condition. As described above, there are known techniques for automatically setting the exposure (face priority AE), adjusting the white balance (face priority AWB), and automatically setting the flash emission conditions. Furthermore, in addition to face detection, a smile detection function for detecting that a subject's face has laughed and an auto shutter function for automatically releasing a shutter when the smile is detected are also known (for example, Patent Documents 1 and 2). reference).

JP 2009-005316 A JP 2009-010776 A

  Such a smile detection function is convenient and practical because of the high frequency of shooting people, but depending on the face and shooting environment, the face or smile may not be detected well, or it is determined that it is laughed. There are also problems such as that it takes a long processing time and misses a photo opportunity due to erroneous detection.

  Even if a smile can be photographed, the photographed image is the same as when the user shot the shutter button while looking at the moment of smile, and the processing is complicated and large-scale. However, there was a problem that there were few merits such as new enjoyment of photography and added value.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an image processing method, and a program capable of reliably photographing a specific state of a subject and providing various added values.

In order to achieve the above object, an image processing apparatus according to the present invention includes a photographing unit that photographs a subject at a predetermined time interval, and an image of the face of the subject based on images that are sequentially photographed during photographing by the photographing unit. A smile detection means for detecting the timing when the facial expression becomes smiley, and the facial expression of the subject becomes a smile from the image being photographed by the photographing means based on the timing detected by the smile detection means. A smile image that is an image taken when the subject is photographed, and a non-smile image that is an image taken before or after the facial expression of the subject becomes a smile, and the identified smile image and non-smile image Recording means for associating and recording a plurality of images including the image of the face of the subject based on the plurality of images recorded by the recording means and including the smile image and the non-smile image A non-smiling image captured before the facial expression of the subject becomes a smile. A first recording / reproducing method for reproducing a smiling image after reproducing a non-smiling image photographed before becoming a smile according to the order in which each image was photographed, or the recording unit However, the non-smiling image taken after the facial expression of the subject becomes a smile is recorded, and the reproduction means switches the order in which each image was taken, and the non-smile taken after the smile Recording and playback are performed by the second recording / playback method of playing back a smile image after playing back an image.
Further, as a preferable aspect, the recording unit specifies an image at a detection timing by the smile detection unit as the smile image, and a position of an image that is in a non-smile state in order from the smile image forward or backward. And the image at the searched position is specified as the non-smile image.
Further, as a preferred aspect, in the case of performing recording and reproduction by the second recording / reproducing method, the recording means determines the position of an image that is in a non-smiling state in order from the specified smiling image. A non-smiling image is identified by searching, and the reproduction means reproduces the face image of the subject in a non-smiling state by reversing the time order in which the smiling image and the non-smiling image were taken. It is characterized in that the image is reproduced so as to continuously change from a smiling state to a smiling state.
Further, as a preferred embodiment, a plurality of images before the subject is in a smiling state are obtained by holding the images photographed at a predetermined time interval by the photographing means in a cyclic manner for a predetermined number or a predetermined time. Holding means for holding the image, and when performing recording and reproduction by the first recording / reproducing method, the recording means further specifies the smile image from the images held by the holding means In addition, it is possible to specify the non-smiling image by searching for the position of the image that is in the non-smiling state in order from the specified smiling image.
Further, as a preferred aspect, the reproducing means is further configured such that, based on the smile image and the non-smile image recorded by the recording means, the facial expression of the subject is the facial expression in the smile image and the non-smile image. The image is reproduced so that the facial expression of the subject continuously changes from a non-smiling state to a smiling state by interpolating and generating an image that is an intermediate facial expression of the facial expression in .

According to another aspect of the present invention, there is provided an image processing apparatus for photographing a subject at a predetermined time interval, and the state of the subject based on images sequentially photographed during photographing by the photographing unit. A detecting unit that detects a timing when the predetermined state is reached, and a state in which the subject is in a predetermined state from among the images being shot by the shooting unit based on the timing detected by the detecting unit; A predetermined state image that is a photographed image and a non-predetermined state image that is an image photographed before or after the state of the subject becomes a predetermined state. Based on a plurality of images including the predetermined state image and the non-predetermined state image recorded by the recording unit that records a plurality of images including the image in association with each other, Reproduction means for reproducing an image so that the state of the subject continuously changes from a non-predetermined state to a predetermined state, and the recording means is a non-photographed image taken before the subject state becomes a predetermined state. First recording / reproduction in which a predetermined state image is recorded, and the reproduction means reproduces the predetermined state image after reproducing the non-predetermined state image photographed before entering the predetermined state according to the order in which each image was photographed. The method or the recording means records a non-predetermined state image taken after the subject has become a predetermined state, and the reproducing means changes the order in which the images were taken, Recording and reproduction by the second recording / reproducing method of reproducing the predetermined state image after reproducing the non-predetermined state image taken after entering the state is characterized.
Further, as a preferred aspect, the recording unit may further include an image group including an image when the subject is in a predetermined state and a plurality of images before or after the subject is in the predetermined state. It is characterized in that an image when the state is reached is recorded in an identifiable file format.

Further, as a preferable aspect , the recording unit further includes an image group including an image when the subject is in a predetermined state and a plurality of images before or after the subject is in the predetermined state. It is recorded in a file format in which the time series relationship can be specified.

In a preferred aspect , the reproducing means further reproduces a plurality of images in the image group recorded by the recording means in the order of time series specified by the file format.

Further, as a preferred aspect, for a plurality of image groups recorded by the recording means, reference images based on images when the subject specified by the file format is in a predetermined state are generated and displayed in a list. Display means; and designation means for designating a series of images to be reproduced by the reproduction means from the list-displayed reference images, wherein the reproduction means includes an image group corresponding to the reference image designated by the designation means The subject is moved from the non-predetermined state to the predetermined state based on an image when the subject in the image group is in a predetermined state and a plurality of images before or after the predetermined state. It is characterized in that an image is reproduced so as to continuously change to a state.

Further, as a preferred aspect, a plurality of images before the subject is in a predetermined state are obtained by cyclically holding a predetermined number of images or a predetermined time for images captured by the imaging unit at a predetermined time interval. Holding means for holding the image, wherein the recording means includes an image when the subject detected by the detection means is in a predetermined state and before the predetermined state held by the holding means. A plurality of images recorded in association with each other, and the reproduction unit records the image when the subject is in a predetermined state and the plurality of images before the predetermined state is recorded, recorded by the recording unit. The image is reproduced so that the subject continuously changes from the non-predetermined state to the predetermined state.

Further, as a preferable aspect, when the detection unit detects that the subject is in a predetermined state, the image when the subject is in the predetermined state and the subject are in the predetermined state. And holding means for holding a plurality of images after the recording, wherein the recording means is held by the holding means and an image when the subject detected by the detecting means is in a predetermined state. A plurality of images after the predetermined state is recorded in association with each other, and the reproduction unit records the image recorded by the recording unit when the subject is in the predetermined state, and the predetermined unit. Image reproduction is performed such that the subject continuously changes from a non-predetermined state to the predetermined state based on a plurality of images after entering the state.

Further, as a preferred aspect, an intermediate image that generates an intermediate state image of each image based on an image when the subject is in a predetermined state and a plurality of images after the predetermined state is reached. And a recording unit configured to generate an image when the subject is in a predetermined state, a plurality of images after the predetermined state is held in the holding unit, and the intermediate image. The intermediate state image generated by the generation unit is recorded in a file format in which a time-series relationship can be specified.

In order to achieve the above object, according to the image processing method of the present invention, the state of the subject is predetermined based on a photographing step of photographing the subject at predetermined time intervals and images sequentially taken during the photographing. And a detection step for detecting the timing when the state of the subject is detected, and based on the detected timing, the image was taken when the subject was in a predetermined state from among the images being photographed by the photographing means and the predetermined state image is an image, the state of the object is to identify the non-predetermined state image that is a captured image before or after a predetermined condition, and the specified predetermined state image and the non-predetermined state image a recording step of recording a plurality of associated images, including, on the basis of the recorded plurality of images including said predetermined state image and the non-predetermined state image, the state of the object is non-plant Look including a reproducing step of image reproduction to continuously change from a state to a predetermined state, the recording step records the non-predetermined state image taken before the state of the object becomes a predetermined state In the first recording / reproducing method, the reproduction step reproduces the predetermined state image after reproducing the non-predetermined state image captured before the predetermined state is reached according to the order in which the images were captured, or the recording The step records a non-predetermined state image taken after the subject is in a predetermined state, and the reproduction step is taken after the order in which each image was taken is changed to the predetermined state. Recording and reproduction by the second recording / reproducing method of reproducing the predetermined state image after reproducing the non-predetermined state image is performed .

In order to achieve the above object, a program according to the present invention is based on a shooting function for shooting a subject at a predetermined time interval on a computer of an image processing apparatus, and images that are sequentially shot during shooting by the shooting function. detection function for detecting the timing at which the state of the object becomes a predetermined state, based on the timing detected by the detecting function, from the image being captured by the imaging function, the state of the object is given and the predetermined state image a taken image when the state, given the state of the object to identify the non-predetermined state image that is a captured image before or after a predetermined state, being this particular recording function for recording a plurality of associated images including the state image and a non-predetermined state image, recorded by the recording function, the predetermined state image and a non-predetermined state image Based on a plurality of images including bets, the continuously image reproduction to vary to execute the reproduction function and the state of an object from a non-predetermined state to a predetermined state, the recording function, the state of the object After recording a non-predetermined state image shot before entering a predetermined state and the playback function playing back the non-predetermined state image shot before entering the predetermined state according to the order in which each image was captured A first recording / reproducing method for reproducing a predetermined state image or the recording function records a non-predetermined state image taken after the state of the subject becomes a predetermined state, and the reproducing function The order in which the images are photographed is switched, and recording and reproduction are performed by a second recording / reproducing method of reproducing the predetermined state image after reproducing the non-predetermined state image captured after the predetermined state is achieved. .

  According to the present invention, there is an advantage that a specific state of a subject can be reliably photographed and various added values can be provided.

It is a block diagram which shows the structure of the digital camera by 1st Embodiment of this invention. It is a block diagram which shows the structure of the image process part of the digital camera by this 1st Embodiment. It is a schematic diagram which shows the external appearance of the digital camera by this 1st Embodiment. 4 is a flowchart for explaining the operation of the digital camera according to the first embodiment. 4 is a flowchart for explaining the operation of the digital camera according to the first embodiment. 4 is a flowchart for explaining the operation of the digital camera according to the first embodiment. It is a conceptual diagram for demonstrating operation | movement of the smile detection imaging | photography mode by the digital camera of the 1st embodiment. It is a conceptual diagram which shows the structure of the picked-up image data by the digital camera of this 1st Embodiment. It is a flowchart for demonstrating operation | movement of the digital camera by this 2nd Embodiment. It is a flowchart for demonstrating operation | movement of the digital camera by this 2nd Embodiment. It is a flowchart for demonstrating operation | movement of the digital camera by this 2nd Embodiment. It is a conceptual diagram for demonstrating operation | movement of the smile detection imaging | photography mode by the digital camera of the 1st embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

A. Configuration of Embodiment FIG. 1 is a block diagram showing a configuration of a digital camera according to an embodiment of the present invention. In the figure, the optical lens 1 is composed of a lens group in which aspherical lenses are stacked. The optical system driving unit 2 performs focusing by driving the lens group. The diaphragm 3 opens and closes under the control of the diaphragm / shutter driving unit 5 to adjust the amount of incident light from the optical lens 1 to the imaging unit 6. The shutter 4 is a so-called mechanical shutter that opens when the release button is operated. The diaphragm / shutter driving unit 5 controls the driving of the diaphragm 3 and the shutter 4.

  The imaging unit 6 includes a CCD and a CMOS, forms a subject image (image), and converts the light intensity of each color of RGB into a current value. The signal processing unit 7 performs correlated double sampling processing and signal amplification processing to suppress noise and color unevenness, and converts the sampled / amplified analog signal into a digital signal (converts each color of RGB and CMYG into 12-bit data). Output to the bus line). The imaging drive unit 8 controls operations of the imaging unit 6 and the signal processing unit 7.

  In the first embodiment, the image processing unit 9 extracts a face area of a person, extracts a facial part and a facial feature from the extracted face area, and detects a face detection / authentication / expression. Various image processing. Details will be described later. The encoding / decoding unit 10 compresses and encodes digital data (uncompressed RAW image data) input from the image processing unit 9 in the JPEG format, or is stored in the image storage unit 11 in the reproduction mode. Stretch the image.

  The image storage unit 11 includes a memory medium 12 and records captured image data. In particular, in the first embodiment, the images taken by the imaging unit 6 are held cyclically while being sequentially updated for a predetermined number or a predetermined time. The display unit 13 includes a color TFT liquid crystal, an STN liquid crystal, or the like, and displays a preview image, image data after shooting (smile video), a setting menu, and the like.

  The photometry unit 14 measures the amount of light. The distance measuring unit 15 measures the distance to the subject. The illumination unit 16 compensates for insufficient exposure. The operation input unit 17 includes a release button and various setting buttons. The control unit 18 executes a program stored in a program memory (not shown) to control the entire digital camera (imaging device).

  The control unit 18 includes a shooting mode selection unit 20, a shooting control unit 21, a condition setting unit 22, a first setting unit 23, and a second setting unit 24. The shooting mode selection unit 20 selects a shooting mode according to a user instruction from the operation input unit 17 and instructs parameters in the shooting control unit 21, condition setting unit 22, first setting unit 23, and second setting unit 24.

  The imaging control unit 21 follows the instruction from the operation input unit 17, the instruction from the imaging mode selection unit 20, the imaging conditions from the condition setting unit 22, etc., the optical system driving unit 2, the aperture / shutter driving unit 5, and the imaging driving unit. 8 is controlled, and photometry is performed by controlling photometry in the photometry unit 14, ranging in the distance measurement unit 15, and illumination by the illumination unit 16.

  The condition setting unit 22 sets an AF frame, a photometric frame, WB (white balance), illumination conditions, and other shooting conditions according to the shooting mode, and supplies them to the shooting control unit 21. The first setting unit 23 sets detection parameters for performing facial part feature detection, face detection / authentication, dictionary data, and the like according to the imaging mode, and supplies them to the image processing unit 9. The second setting unit 24 sets parameters for image correction / combination / processing and supplies them to the image processing unit 9.

Next, FIG. 2 is a block diagram showing the configuration of the image processing unit according to the first embodiment.
The image processing unit 9 includes an image buffer memory 30, a preprocessing / binarization unit 31, a face area detection unit 32, a face part / feature extraction unit 33, a feature extraction dictionary 34, a face detection / authentication / expression detection unit 35, A face detection / authentication dictionary 36 and an image correction / combination processing unit 37 are provided.

  The image buffer memory 30 temporarily holds the image data from the signal processing unit. The preprocessing / binarization unit 31 performs preprocessing for detecting a face area on the image data and binarizes the image data. The face area detection unit 32 detects a face area from the binarized image data. The face part / feature extraction unit 33 extracts a face part (eyes, nose, mouth, etc.) and features according to the face feature data stored in the feature extraction dictionary 34. The face detection / authentication / expression detection unit 35 detects what expression (specifically a smile) the face has in accordance with the face detection / authentication data stored in the face detection / authentication dictionary 36.

  The image correction / combination processing unit 37 includes a lightness / hue / saturation / tone correction unit 40, a beautiful skin / face processing unit 41, a pseudo makeup processing unit 42, a pseudo shaping processing unit 43, and a facial expression deformation processing unit 44. . The lightness / hue / saturation / tone correction unit 40 corrects the lightness, hue, saturation, and gradation of the image. The beautiful skin / face processing unit 41 performs skin care / facial processing on the face region extracted by the face region extraction unit 32.

  The pseudo makeup processing unit 42 performs the pseudo makeup processing on the face area that has been subjected to the skin beautification / face treatment. The pseudo shaping processing unit 43 performs a pseudo shaping process on the face area that has been subjected to the pseudo makeup process. The facial expression deformation processing unit 44 performs facial expression deformation processing (deforms non-smile to smile) on the face area that has been subjected to the pseudo shaping processing.

Next, FIG. 3 is a schematic diagram showing an appearance of the digital camera according to the first embodiment.
A strobe 50 and a light receiving window 51 of the photographic optical system are arranged in front of the digital camera. In addition, a shutter switch (release button) 52, a power switch 53, and the like are disposed on the top of the digital camera. Inside the digital camera, a mirror (or prism) 54 that reflects incident light from the light receiving unit 51 downward by 90 degrees is provided, and the lens group 55 of the photographing optical lens 1 and the imaging unit 6 are disposed on the optical axis thereof. Has been placed.

  Also, on the back of the digital camera, there is a display unit 13 as an electronic viewfinder / monitor, a mode changeover switch 56 for switching shooting modes, zoom operation keys for zoom operations (in addition to the functions, index display / playback display keys can also be used) 57), cursor key (next / previous image display key) 58, enter / OK key 59, menu key 60 for calling a menu screen for various settings, display key (face search key) 61, memory medium / battery A storage unit 62 and the like are disposed.

Next, the operation of the first embodiment will be described.
4 to 6 are flowcharts for explaining the operation of the digital camera according to the first embodiment. FIG. 7 is a conceptual diagram for explaining the operation of the smile detection photographing mode by the digital camera of the first embodiment. In the first embodiment, detection of a human face area, detection of a smile, and the like are performed. However, since these techniques are well-known techniques, the description thereof is omitted.

  When the shooting control process is executed, it is first determined whether or not the shooting mode is set (step S10). If it is the shooting mode, various shooting modes, scene modes, etc. are selected according to the user operation, and shooting conditions, AF frames, etc. are set (step S12). Next, the through image of the subject image is displayed on the display unit (step S14), and it is determined whether or not the mode is the person shooting mode and the face detection shooting mode (step S16).

  If the mode is the person shooting mode or the face detection shooting mode, face detection processing is executed (step S18), it is determined whether a face area is detected (step S20), and the face area is detected. If it is detected, shooting conditions such as an AF frame are set based on the detected face area, and the face area is distinguished and displayed (blue frame, red frame, etc.) (step S22). Next, it is determined whether or not the release button has been half-pressed (step S24). If the release button has not been half-pressed, the process returns to step S14, and steps S14 to S24 described above are performed until the release button is half-pressed. repeat.

  On the other hand, when neither the portrait shooting mode nor the face detection shooting mode is performed, it is determined whether or not the release button has been pressed halfway without performing the process related to the face detection process (step S24). In this case, the process returns to step S14, and steps S14 to S24 described above are repeated until the release button is pressed halfway.

  When the release button is half-pressed, AF processing is performed so that the set AF frame (or the detected face area) falls within the depth of field, and AE processing is performed so as to satisfy the appropriate exposure condition (step S26). It is determined whether the smile auto shutter function is set to ON (step S28). The smile auto-shutter function is a function for automatically taking a picture with a shutter automatically released when a smile is detected.

  If the smile auto shutter function is not set to ON, it is determined whether or not the release button has been fully pressed (step S36). If the release button has not been fully pressed, the process returns to step S28 to release the release button. Steps S28 and S36 are repeatedly executed until the button is fully pressed. When the release button is fully pressed, the exposure and shooting process is performed under the set and adjusted shooting conditions (step S38).

  On the other hand, when the smile auto shutter function is set to ON, the PAST continuous shooting function is activated, and automatically continuously shot images are sequentially and sequentially recorded in a temporary memory (step S30). That is, as shown in FIG. 7, a predetermined number (or a predetermined time) of images IM1, IM2, IM3, IM4,... Are sequentially recorded in the temporary memory. The continuously photographed images are sequentially updated while the release button is half-pressed, and a predetermined number of images (or a predetermined time) are always held.

  Next, a smile detection process is performed from the photographed image (step S32), and it is determined whether a smile is detected (step S34). If no smile is detected, it is determined whether or not the release button has been fully pressed (step S36). If no smile is detected and the release button is not fully pressed, the process returns to step S28, and PAST continuous shooting and smile detection are continued.

  On the other hand, if a smile is detected or no smile is detected, but the release button is fully pressed, the exposure and shooting process is performed under the set and adjusted shooting conditions (step S38). FIG. 7 shows a case where a smile is detected, and a smile image (target image) TIM is photographed by the main photographing process.

  In either case, PAST continuous shooting is stopped after a predetermined time of the exposure & photographing process (step S40), and it is determined whether or not the smile auto shutter function is set to ON (step S42). If the smile auto shutter function is not set to ON, the captured image is encoded / compressed and stored in the memory medium (step S44). Next, only the actual photographed image is displayed for review (step S46).

  On the other hand, when the smile auto shutter function is set to ON, several frames of non-smile images are searched from the images before and after the PAST continuous shooting and accumulated (step S48: refer to the third row from the top in FIG. 7). Then, it is determined whether or not the search is successful (step S50). If the retrieval of the non-smiling image fails, the main image is encoded / compressed and stored in the memory medium (step S44), and only the main image is reviewed and displayed (step S46). Thereafter, the process returns to the main routine.

  When searching (extracting) several frames of non-smiling images to be recorded as smile videos from all the images accumulated by PAST continuous shooting, search (extract) by skipping at regular intervals. Make sure that the video recording capacity does not become too large. However, when the shooting interval by PAST continuous shooting is long, all images accumulated by PAST continuous shooting may be recorded as a smile video.

  On the other hand, when the retrieval of the non-smiling image is successful, as shown in the fourth row from the top in FIG. 7, the smiling image (main image) of the main shooting and several frames of the searched PAST shooting image ( Sub-picture) is encoded / compressed and associated and stored in a memory medium in MPF (Multi Picture Format) format or the like (step S52: refer to the third row from the top in FIG. 7). The association file includes SOI (image data start), main image header, main image compressed data, EOI (image data end), SOI (image data start), sub image (1) header, and sub image (1) compression. 1 from SOI to EOI, such as data, EOI (image data end),..., SOI (image data start), sub-image (n) header, sub-image (n) compressed data, EOI (image data end) Corresponding to two images, the captured images are arranged in reverse order to the captured time axis. Note that details of the MPF format in which the smile image of the actual photographing and the several frames of the searched PAST captured images are stored in association with each other will be described later.

  Next, the smile image (or non-smile image) of the actual shooting and several frames of the non-smile image (or smile image) of the PAST shooting are constant from the non-smile image (n) to the smile image (0). The display is switched sequentially for each time to display the review (smile video display: the bottom row in FIG. 7) (step S54).

  On the other hand, if the shooting mode is not set in step S10, it is determined whether or not the playback / editing mode is set (step S56). If not in the playing / editing mode, the process proceeds to other mode processing (step S58). ). On the other hand, if the playback / editing mode is selected, it is determined whether or not the thumbnail list or the playback image is selected (step S60). Proceed to the editing process (step S62).

  In the case of selection of a thumbnail list or a playback image, an image to be played back is selected according to a user operation (step S64), and it is determined whether the image is a smile movie setting image (step S66). ). If the selected playback image is not a smile video setting image, the selected image is read out and subjected to signal processing (step S68), and the selected image is played back and displayed (step S70). Thereafter, the process returns to the main routine.

  On the other hand, if the selected reproduction image is an image set with a smile movie, the selected image and the sub-image (of the smile movie) stored in association with each other are read out and subjected to a decoding process (step S72). Next, in steps S74 to S80, a smile video display is performed in which the non-smiling images are displayed in order, and the smiling image is displayed last.

  That is, the main image / sub-image of the selected image is displayed (step S74), and it is determined whether or not the display time of each frame has ended (step S76). If the display time has not ended, the process returns to step S74. When the display of the main image / sub-image of the selected image is continued and the display time ends, it is determined whether or not the predetermined elapsed time has ended (step S78). Switches to the next main image / sub image (step S80), returns to step S74, and displays the next main image / sub image. Thereafter, until the predetermined elapsed time ends, a smile video display is performed in which the non-smiling image is displayed in order and the smiling image is displayed last. When the predetermined elapsed time ends, the process returns to the main routine.

  Here, FIG. 8 is a conceptual diagram showing an example of the MPF format in the present embodiment in which the smile image of the actual shooting and the several frames (non-smile images) of the searched PAST shot image are stored and recorded in association with each other. FIG. The MPF format file is composed of a single block consisting of SOI (image data start), header, image data, and EOI (image data end) as a single block. In the first embodiment, first, the main image (smile image) is displayed in the first block, and then the non-smile image (sub-image (1)), non-smile image (sub-image (2), ... and recorded so as to be a non-smiling image (sub-image (n)).

  The header of each block includes a thumbnail of the corresponding image. Specifically, the header includes an APP1 Marker, an APP1 Length, an Exif identification code, main image / sub-image information, and main image / sub-image thumbnails. The main image / sub image information corresponds to Exif IFD (version information tag, image characteristic information tag, structure information tag,... Sub image area presence / absence information,...), Value of Exif IFD, and a plurality of sub images. A sub-image index IFD and a sub-image index Value of IFD are included.

  In the first embodiment, the structure of the MPF format is used to store and record the smile image of the actual shooting and the several frames (non-smile images) of the searched PAST shot images. However, the present invention is not limited to this, and any other file format may be used as long as it is a structure in which the association between the smile image of the actual photographing and the plurality of non-smile images can be understood.

  According to the first embodiment described above, in the smile shooting process, face detection and smile detection are performed to perform shooting in a smiling state, and in parallel, images from PAST continuous shooting to smile shooting are performed. By continuously shooting (non-smiling images) and sequentially recording them in a temporary storage memory in a cyclic manner, the smiling images and the non-smiling images within a predetermined time until reaching the smiling images are stored and recorded in association. Sometimes, according to the associated image stored and recorded, the change in facial expression at the time of laughing can be more effectively reproduced by sequentially switching and displaying the review so as to end with a smile image for the first time from a non-smile image.

  For this reason, the facial expression of the subject person can be reproduced with a more realistic, clearer, and impressive expression, so that it is possible to further enhance entertainment and added value such as smile detection and auto shutter function. . In addition, when shooting a baby or a child's smile, it is possible to further enhance commemoration, added value as a record, and excitement.

  In addition, since the image for the smile movie is also recorded and saved at the same time as the normal actual image, the actual image can be selected and printed as a still image as usual, or in the MPF format for the smile movie. Video animations, video playback, and short clip smile videos can be uploaded to the network (web) as public images such as blogs, avatars, icons, etc. The usage, appreciation method, and expression can be expanded accordingly.

B. Second Embodiment Next, a second embodiment of the present invention will be described.
In the second embodiment, in addition to the main shooting, automatic shooting is continued from the main shooting until the elapse of a predetermined time (about 0.1 to 1 second later), and the first frame to the second frame after the main shooting smile shooting. About two or three images are taken continuously together with the main image. The configuration of the digital camera is the same as that shown in FIGS.

Next, the operation of the second embodiment will be described.
9 to 11 are flowcharts for explaining the operation of the digital camera according to the second embodiment. FIG. 12 is a conceptual diagram for explaining the operation of the smile detection photographing mode by the digital camera of the second embodiment.

  When the shooting control process is executed, it is first determined whether or not it is a shooting mode (step S100). If it is a shooting mode, various shooting modes, scene modes, and the like are selected according to a user operation. Then, shooting conditions, an AF frame, etc. are set (step S102), a through image of the subject image is displayed on the display unit (step S104), and whether or not it is a person shooting mode or a face detection shooting mode. Is determined (step S106).

  If it is the person shooting mode or the face detection shooting mode, face detection processing is executed (step S108), it is determined whether a face area is detected (step S110), and the face area is detected. If it is detected, shooting conditions such as an AF frame are set based on the detected face area, and the face area is displayed in a distinguishable manner (blue frame, red frame, etc.) (step S112). Next, it is determined whether or not the release button has been half-pressed (step S114). If the release button has not been half-pressed, the process returns to step S104, and steps S104 to S114 described above are performed until the release button is half-pressed. repeat.

  On the other hand, when neither the person shooting mode nor the face detection shooting mode is used, it is determined whether or not the release button has been pressed halfway without performing the process related to the face detection process (step S114). In this case, the process returns to step S104, and the above-described steps S104 to S114 are repeated until the release button is pressed halfway.

  When the release button is half-pressed, AF processing is performed on the set AF frame (or detected face area) and AE processing is performed so that the appropriate exposure condition is satisfied (step S116), and the smile auto shutter function is turned on. It is determined whether it is a setting (step S118). If the smile auto shutter function is not set to ON, it is determined whether or not the release button has been fully pressed (step S124). If the release button has not been fully pressed, the process returns to step S118 to release. Steps S118 and S124 are repeatedly executed until the button is fully pressed, and when the release button is fully pressed, exposure and shooting processing is performed under the set and adjusted shooting conditions (step S126).

  On the other hand, when the smile auto shutter function is set to ON, smile detection processing is performed from the captured image (step S120), and it is determined whether a smile is detected (step S122). If no smile is detected, it is determined whether or not the release button has been fully pressed (step S124). If no smile is detected and the release button is not fully pressed, the process returns to step S118 and the smile detection is continued.

  On the other hand, if a smile is detected or no smile is detected, but the release button is fully pressed, the exposure and shooting process is performed under the set and adjusted shooting conditions (step S126). FIG. 12 shows a case where a smile is detected, and a smile image (target image) TIM is captured by the actual capturing process.

  Next, it is determined whether or not the smile auto shutter function is set to ON (step S128). If the smile auto shutter function is not set to ON, the actual captured image is encoded / compressed and stored in a memory medium (step S130), and only the actual captured image is displayed for review (step S132). Thereafter, the process returns to the main routine.

  On the other hand, when the smile auto shutter function is set to ON, the timer is counted (step S134), and it is determined whether or not a predetermined time (about 0.1 to 1 second) has elapsed (step S136). If the predetermined time has not elapsed, the process returns to step S134 to continue the timer timing. When the time measured by the timer has elapsed for a predetermined time (about 0.1 to 1 second), exposure and shooting processing is performed (step S138), and it is determined whether or not a predetermined number of frames (1 to several frames) has been shot. (Step S140).

  If the predetermined number of frames has not been shot, the process returns to step S134, and steps S134 to S140 are repeated again. As a result, after the main shooting (smiling shooting), a predetermined number of frames (1 to several frames) are automatically shot, and several (2 to 3) images IM1 and IM2 are combined with the main shooting image TIM. Are continuously photographed (see the top row in FIG. 12).

  Then, when shooting of the predetermined number of frames is completed, as shown in FIG. 12, based on the smile image (main image) of the main shooting and the subsequent automatic shooting image (1 to several frames) image (sub-image). The intermediate images MIM1 and MIM2 ranging from the non-smiling image to the smiling image are synthesized by morphing or warping processing (step S144).

  Next, the actual captured image TIM, the automatically captured images (one to several frames) IM1 and IM2, and the combined intermediate images MIM1 and MIN2 are encoded / compressed and stored in a memory medium in association with each other, or the actual captured image Separately, it is recorded as a moving image file (step S146: refer to the second row from the bottom in FIG. 12). Next, the main photographed image TIM, the automatic photographed images IM1 and IM2, and the composite intermediate images MIM1 and MIN2 are sequentially switched from non-smile to smile to display a review (smile movie display: bottom in FIG. 12) ( Step S146). Thereafter, the process returns to the main routine.

  On the other hand, if the shooting mode is not set in step S100, it is determined whether or not the playback / editing mode is set (step S148). If not the playback / editing mode, the process proceeds to other mode processing (step S150). ). On the other hand, if the playback / editing mode is selected, it is determined whether or not the thumbnail list or the playback image is selected (step S152). Proceed to the editing process (step S154).

  If the thumbnail list or the playback image is selected, an image to be played back is selected according to the user's operation (step S156), and it is determined whether the image is a smile movie setting image (step S158). ). If the selected replay image is not a smile video setting image, the selected image is read and subjected to signal processing (step S160), and the selected image is replayed and displayed (step S162). Thereafter, the process returns to the main routine.

  On the other hand, if the selected playback image is a smile video setting image, the selected image and the sub-image (of the smile video) or the moving image file stored in association with each other are read out and subjected to a decoding process (step S1). S164). Next, in steps S166 to S172, a smile video display is performed in which the non-smiling image is displayed in order, and the smiling image is displayed last.

  That is, the main image / sub-image of the selected image is displayed (step S166), and it is determined whether the display time of each frame has ended (step S168). If the display time has not ended, the process returns to step S166. When the display of the main image / sub-image of the selected image is continued and the display time ends, it is determined whether or not the predetermined elapsed time has ended (step S170). Switches to the next main image / sub-image (step S172), returns to step S166, and displays the next main image / sub-image. Thereafter, until the predetermined elapsed time ends, a smile video display is performed in which the non-smiling image is displayed in order and the smiling image is displayed last. When the predetermined elapsed time ends, the process returns to the main routine.

  According to the second embodiment described above, since there is a high probability that an image taken after a predetermined time has passed since the main shooting has taken a non-smiling image, such a non-smiling image and a main shooting Based on a smile image, the intermediate image is automatically generated by using a morphing intermediate image composition or warping deformation composition, and a combined image representing facial expression changes from non-smile to smile Is generated and stored in a memory medium in association with the actual captured image, so that at the time of playback, in accordance with the stored and recorded associated image, a review image is displayed by sequentially switching from a non-smile image to a smile image for the first time. This makes it possible to more effectively reproduce facial expression changes when laughing.

  For this reason, the facial expression of the subject person can be reproduced with a more realistic, clearer, and impressive expression, so that it is possible to further enhance entertainment and added value such as smile detection and auto shutter function. . In addition, when shooting a baby or a child's smile, it is possible to further enhance commemoration, added value as a record, and excitement.

  In addition, since the image for the smile movie is also recorded and saved at the same time as the normal actual image, the actual image can be selected and printed as a still image as usual, or in the MPF format for the smile movie. Video animations, video playback, and short clip smile videos can be uploaded to the network (web) as public images such as blogs, avatars, icons, etc. The usage, appreciation method, and expression can be expanded accordingly.

  In the first and second embodiments described above, the smile of a person is used as the target image. However, the present invention is not limited to this, and may be an angry face or a crying face. In the first and second embodiments described above, the subject is a person. However, the subject is not limited thereto, and the subject may be an animal (dog, cat, bird, etc.). In this case, the subject state (target image) may be a running state for a dog or a cat, a jumping state, or a flying state for a bird.

  In the first and second embodiments described above, a case where a person is a subject and a smile is the subject state has been described. However, as described above, when there are a plurality of subjects and subject states, the user can arbitrarily select the subject. May be specified.

  In the first and second embodiments described above, the smile image and the non-smile image of the actual photographing are used in a file format such as the MPF format, but Motion JPEG, MPEG4, H.264, etc. It may be stored and recorded as moving image data in a file format such as H.264 standard.

DESCRIPTION OF SYMBOLS 1 Image | photographing optical system 2 Optical system drive part 3 Aperture 4 Shutter 5 Aperture / shutter drive part 6 Imaging part 7 Signal processing part 8 Imaging drive part 9 Image processing part 10 Encoding / decoding part 11 Image storage part 12 Memory medium 13 Display Section 14 Photometry section 15 Distance measurement section 16 Illumination section 17 Operation input section 18 Control section 20 Shooting mode selection section 21 Shooting control section 22 Condition setting section 23 First setting section 24 Second setting section 30 Image buffer memory 31 Pre-processing / second Numeralization unit 32 Face area detection unit 33 Face part / feature extraction unit 34 Feature extraction dictionary 35 Face detection / authentication / expression detection unit 36 Face detection / authentication dictionary 37 Image correction / combination processing unit 40 Brightness / Hue / Saturation / Tone correction unit 41 Beautiful skin / face processing unit 42 Pseudo makeup processing unit 43 Pseudo shaping processing unit 44 Facial expression deformation processing unit

Claims (15)

Photographing means for photographing a subject at predetermined time intervals;
Smile detection means for detecting the timing when the facial expression of the subject becomes a smile based on images sequentially taken during shooting by the shooting means;
Based on the timing detected by the smile detection means, a smile image, which is an image taken when the facial expression of the subject becomes a smile, among the images being shot by the shooting means, and the subject Recording means for identifying a non-smiling image that is an image taken before or after the facial expression of the face becomes a smile and associating and recording a plurality of images including the identified smiling image and the non-smiling image;
Based on a plurality of images including the smile image and the non-smile image recorded by the recording means, the image is such that the facial expression of the subject continuously changes from a non-smile state to a smile state. for example Bei and reproducing means for reproducing,
The recording means records a non-smiling image taken before the facial expression of the subject becomes a smile, and the reproducing means follows the order in which the images were taken before A first recording / reproducing method for reproducing a smiling image after reproducing a smiling image, or the recording unit records a non-smiling image taken after the facial expression of the subject becomes a smiling face, and the reproducing unit However, the order in which each image was photographed is changed, and recording and reproduction are performed by the second recording / reproducing method of reproducing a smiling image after reproducing a non-smiling image taken after a smile. Image processing device.   The recording unit specifies an image at a detection timing by the smile detection unit as the smile image, and searches for a position of an image that is in a non-smiling state in order from the smile image to the front or the back. The image processing apparatus according to claim 1, wherein an image at a selected position is specified as the non-smiling image. When performing recording and reproduction by the second recording / reproducing method,
The recording means specifies a non-smiling image by searching for a position of an image that is in a non-smiling state in order from the identified smiling image,
The reproduction means continuously changes the facial expression of the subject from a non-smile state to a smile state by reversing the time order in which the smile image and the non-smile image were taken. The image processing apparatus according to claim 2, wherein the image is reproduced as described above. Holding means for holding a plurality of images before the subject is in a smiling state by holding the images photographed by the photographing means at predetermined time intervals in a cyclic manner for a predetermined number or a predetermined time. Prepared,
When performing recording and reproduction by the first recording / reproducing method,
The recording means further specifies the smile image from the images held in the holding means, and searches for the position of the image that is in a non-smile state in order from the specified smile image. The image processing apparatus according to claim 2, wherein a non-smiling image can be specified.   Based on the smile image and the non-smile image recorded by the recording unit, the reproduction unit has a facial expression of the subject intermediate between the facial expression in the smile image and the facial expression in the non-smile image. The image is reproduced so that the facial expression of the subject continuously changes from a non-smiling state to a smiling state by interpolating and generating an image that becomes a facial expression. An image processing apparatus according to claim 1. Photographing means for photographing a subject at predetermined time intervals;
Detecting means for detecting timing at which the state of the subject is in a predetermined state based on images that are sequentially photographed during photographing by the photographing means;
Based on the timing detected by the detection unit, a predetermined state image that is an image captured when the subject is in a predetermined state from among images being captured by the photographing unit, and the subject Recording means for specifying a non-predetermined state image that is an image taken before or after the state becomes a predetermined state and associating and recording a plurality of images including the specified predetermined state image and the non-predetermined state image When,
Based on a plurality of images including the predetermined state image and the non-predetermined state image recorded by the recording unit, the image is reproduced so that the state of the subject continuously changes from the non-predetermined state to the predetermined state. Reproduction means
With
The recording means records a non-predetermined state image taken before the subject is in a predetermined state, and the reproducing means takes a picture before the predetermined state is reached according to the order in which the images were taken. A first recording / reproducing method for reproducing a predetermined state image after reproducing the non-predetermined state image, or a non-predetermined state image captured after the recording means has become a predetermined state. According to a second recording / reproducing method for recording and reproducing the predetermined state image after the non-predetermined state image that has been captured after the image has been taken in a predetermined state by changing the order in which the images were taken An image processing apparatus that performs recording and reproduction. The recording means includes
An image group including an image when the subject is in a predetermined state and a plurality of images before or after the predetermined state can be identified as an image when the subject is in the predetermined state 7. The image processing apparatus according to claim 6 , wherein recording is performed in a file format. The recording means includes
An image group including an image when the subject is in a predetermined state and a plurality of images before or after the predetermined state is recorded in a file format in which the time-series relationship of each image can be specified The image processing apparatus according to claim 6 , wherein the image processing apparatus is an image processing apparatus. The reproducing means includes
9. The image processing apparatus according to claim 6 , wherein a plurality of images in the image group recorded by the recording unit are reproduced in the order of time series specified by the file format. Display means for generating and displaying a list of reference images based on images when the subject specified by the file format is in a predetermined state for a plurality of image groups recorded by the recording means;
Designating means for designating a series of images to be reproduced by the reproducing means from the list-displayed reference images;
The reproducing means includes
Targeting an image group corresponding to the reference image specified by the specifying means, an image when the subject in the image group is in a predetermined state and a plurality of images before or after the predetermined state is reached The image processing apparatus according to claim 7 , wherein the image is reproduced so that the subject continuously changes from the non-predetermined state to the predetermined state. Holding means for holding a plurality of images before the subject is in a predetermined state by holding the images photographed by the photographing means at predetermined time intervals in a cyclic manner for a predetermined number or a predetermined time. Prepared,
When performing recording and reproduction by the first recording / reproducing method,
The recording means includes
An image when the subject detected by the detection unit is in a predetermined state and a plurality of images held in the holding unit before the predetermined state are recorded in association with each other.
The reproducing means includes
The subject is changed from the non-predetermined state to the predetermined state based on the image recorded by the recording unit when the subject is in a predetermined state and a plurality of images before the predetermined state is reached. The image processing apparatus according to any one of claims 6 to 10, wherein the image is reproduced so as to change continuously. When the detection unit detects that the subject is in a predetermined state, an image when the subject is in a predetermined state and a plurality of images after the subject is in a predetermined state Further comprising holding means for holding
The recording means includes
An image when the subject detected by the detection unit is in a predetermined state and a plurality of images after the predetermined state held by the holding unit are recorded in association with each other.
The reproducing means includes
The subject is changed from the non-predetermined state to the predetermined state based on the image recorded by the recording unit when the subject is in a predetermined state and a plurality of images after the predetermined state is reached. The image processing apparatus according to claim 6, wherein the image reproduction is performed so as to continuously change.
An intermediate image generating means for generating an intermediate state image of each image based on an image when the subject is in a predetermined state and a plurality of images after the predetermined state;
The recording means includes
An image when the subject is in a predetermined state, a plurality of images after the predetermined state is held in the holding unit, and an image in the intermediate state generated by the intermediate image generating unit the image processing apparatus according to claim 12, preparative, time sequences, characterized in that the recording in identifiable file format. A shooting step of shooting a subject at predetermined time intervals;
A detecting step of detecting a timing at which the state of the subject is in a predetermined state based on images that are sequentially captured during the photographing;
Based on the detected timing, a predetermined state image, which is an image captured when the state of the subject becomes a predetermined state , among the images being captured by the photographing unit, and the state of the subject identify non predetermined state image that is a captured image before or after a predetermined condition, and a recording step of recording a plurality of associated images and a the specified predetermined state image and the non-predetermined state image,
A reproduction step of reproducing an image based on a plurality of images including the recorded predetermined state image and non- predetermined state image so that the state of the subject continuously changes from the non- predetermined state to the predetermined state ; seen including,
The recording step records a non-predetermined state image taken before the subject is in a predetermined state, and the reproduction step is taken before the predetermined state is reached according to the order in which the images were taken. A first recording / reproducing method for reproducing a predetermined state image after reproducing the non-predetermined state image, or a non-predetermined state image captured after the recording step has become a predetermined state. According to a second recording / reproducing method for recording and reproducing the predetermined state image after reproducing the non-predetermined state image that has been photographed after changing the order in which the respective images were photographed and entering the predetermined state An image processing method comprising performing recording and reproduction . In the computer of the image processing device,
Shooting function for shooting the subject at predetermined time intervals,
A detection function for detecting a timing when the state of the subject is in a predetermined state based on images that are sequentially captured during shooting by the shooting function;
Based on the timing detected by the detection function, a predetermined state image which is an image captured when the state of the subject becomes a predetermined state from among images being captured by the photographing function, and the subject A recording that identifies a non- predetermined state image that is an image taken before or after the state becomes a predetermined state, and records a plurality of images including the specified predetermined state image and the non- predetermined state image in association with each other function,
Based on a plurality of images recorded by the recording function including the predetermined state image and the non- predetermined state image, the image is reproduced so that the state of the subject continuously changes from the non- predetermined state to the predetermined state . Execute playback function ,
The recording function records a non-predetermined state image taken before the subject is in a predetermined state, and the playback function takes a picture before the predetermined state is reached according to the order in which the images were taken. A first recording / reproducing method for reproducing a predetermined state image after reproducing the non-predetermined state image, or a non-predetermined state image captured after the recording function is changed to a predetermined state. According to a second recording / reproducing method for recording and reproducing the predetermined state image after reproducing the non-predetermined state image photographed after the predetermined state is changed by changing the order in which each image was photographed. A program characterized by recording and reproduction .

Images