JP2011109271A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus which enables a user to enjoy various variations in state of an object by using a plurality of consecutively photographed images. <P>SOLUTION: The image processing apparatus includes: a storage unit where a consecutive image group consisting of a plurality of consecutively photographed images and information associated with each of the plurality of images are stored; a display unit which is capable of consecutively displaying the consecutive image group stored in the storage unit, in the photographing order; a face detection unit which detects identical faces included in the plurality of images stored in the storage unit; an image correction module which corrects respective images of identical faces detected by the face detection unit so that the identical faces are in same positions in respective images and have the same size; and a display control unit which controls the display unit so that the display unit displays a corrected image group consisting of images resulting from correcting respective images by the image correction module or the consecutive image group. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that performs processing on an image.

  2. Description of the Related Art Conventionally, in imaging devices such as digital cameras and digital video cameras, techniques for performing various image processing on captured images are known. For example, a technique is known that automatically deletes an image that has been shaken due to camera shake of a photographer or movement of a subject (see, for example, Patent Document 1). In this technique, a high frequency component of a facial feature part included in a luminance signal of a subject image is searched, and an image having the high frequency component less than a certain level is regarded as a blurred image and is erased.

  Further, for each of a plurality of time-lapse image frames, an image position where the overlap of two image frames adjacent to each other at the shooting time is maximized is determined, and an output position of each image frame is determined based on the determined image position. Is known (see, for example, Patent Document 2). In this technique, the image position where the overlap is maximized is determined by calculating the other moving direction with respect to one of two image frames adjacent to each other at the photographing time.

Japanese Patent No. 4063592 JP 2005-267005 A

  By the way, when a person is photographed or continuously shot with a moving image, there is a desire to enjoy changes in the facial expression and movement of the person. However, for example, when attention is paid to changes in a person's facial expression, the person's face is enlarged and photographed, so the change in the person's movement must be difficult to understand. When attention is paid to changes in a person's movement, the body of the person is photographed, so the proportion of the face in the entire screen is reduced, making it difficult to detect changes in facial expressions. As described above, in the conventional image processing technology, the user cannot enjoy various state changes of the subject using a plurality of images taken continuously.

  The present invention has been made in view of the above, and provides an image processing apparatus that allows a user to enjoy various state changes of a subject using a plurality of images that are continuously captured. Objective.

  In order to solve the above-described problems and achieve the object, an image processing apparatus according to the present invention stores a continuous image group composed of a plurality of images taken continuously and information related to each of the plurality of images. A display unit capable of continuously displaying the continuous image group stored in the storage unit in order of photographing, and face detection for detecting the same face included in each of the plurality of images stored in the storage unit A correction unit configured to correct each image so that the position and size of each image of the same face detected by the face detection unit are aligned, and a correction image of each image corrected by the image correction unit A display control unit that performs control to display one of the image group and the continuous image group on the display unit.

  In the image processing apparatus according to the present invention as set forth in the invention described above, the image correction unit determines a comparison reference image from the plurality of images, and the size and position of the same face are compared with the comparison reference image. Image correction is performed on the changed image.

  In the image processing apparatus according to the present invention, in the above invention, the image correction unit includes a trimming region determination unit that determines a trimming region in each of the plurality of images, and a trimming region that is determined by the trimming region determination unit. A trimming unit that trims each image in response to the image, and an image enlarging unit that generates the corrected image by enlarging the image trimmed by the trimming unit to the same size as the plurality of images. .

  In the image processing apparatus according to the present invention as set forth in the invention described above, the image correction means may provide an enlargement ratio given by a quotient obtained by dividing the maximum value of the length of the same face in a predetermined direction by the length of each image. An image enlarging unit for enlarging each image, a trimming region determining unit for determining a trimming region in the image enlarged by the image enlarging unit, and trimming each image according to the trimming region determined by the trimming region determining unit And a trimming unit for generating the corrected image.

  In the image processing apparatus according to the present invention, in the above invention, the image enlargement unit may calculate a quotient obtained by dividing a maximum value of the length of the same face in a predetermined direction by a minimum value of the length from a reference enlargement ratio. If larger, each image is magnified with a product obtained by multiplying the magnification by a constant smaller than 1 as a true magnification.

  In the image processing apparatus according to the present invention, the aspect ratio of the trimming area is equal to the aspect ratio of the plurality of images.

  In the image processing apparatus according to the present invention, in the above invention, the corrected image is reduced, and the reduced corrected image is arranged in a partial area of the image before correction to generate a composite image. A synthesizer is provided.

  In the image processing apparatus according to the present invention as set forth in the invention described above, the image correction unit may include a region extraction unit that extracts a region that is common to all of the plurality of images and does not include the same face. And the image composition unit arranges the corrected image corresponding to each image in a region extracted by the region extraction unit of each image.

  In the image processing apparatus according to the present invention as set forth in the invention described above, the image correction unit may change the expression of the same face between two images of which the shooting times are adjacent among the plurality of images. A change detection unit that detects based on a displacement amount of the feature point of the image, and the image composition unit has a large change in facial expression when the change in facial expression detected by the change detection unit is greater than a reference. An image indicating this is synthesized with an image taken later of the two images.

  In the image processing apparatus according to the present invention, in the above invention, the display control unit adds a composite image group composed of composite images of the images synthesized by the image composition unit to the corrected image group and the continuous image group. Control is performed to display any one of the three image groups on the display unit.

  The image processing apparatus according to the present invention further includes an imaging unit that collects light from a predetermined area, images a subject in the area, and generates electronic data of the captured image. The storage unit stores a plurality of captured images continuously captured by the imaging unit, and the display control unit stores the three image groups after the imaging unit finishes capturing the plurality of captured images. The display unit is controlled to display any one of the image groups.

  The image processing apparatus according to the present invention further includes an imaging unit that collects light from a predetermined area, images a subject in the area, and generates electronic data of the captured image. The storage unit stores a plurality of captured images continuously captured by the imaging unit, and the display control unit displays the display while the imaging unit is capturing the plurality of captured images. The control unit sequentially controls the correction image or the composite image corresponding to the latest photographed image.

  According to the present invention, the same face included in each of a plurality of images constituting a continuous image group captured continuously is detected, and the size and position of each detected image of the same face are aligned. Since the correction image group is generated by correcting the image, and the correction image group or the continuous image group including the generated correction image can be displayed, the user can obtain a plurality of images captured continuously. Using these images, you can enjoy various changes in the state of the subject.

FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 1 of the present invention. FIG. 2 is a diagram showing a continuous image group captured by the image processing apparatus according to Embodiment 1 of the present invention. FIG. 3 is a diagram illustrating a trimming region in a captured image. FIG. 4 is a diagram showing a display example on the display unit when the image processing apparatus according to Embodiment 1 of the present invention is set to the expression reproduction mode. FIG. 5 is a diagram showing a display example on the display unit when the image processing apparatus according to Embodiment 1 of the present invention is set to the composite display mode. FIG. 6 is a flowchart showing an outline of processing performed by the image processing apparatus according to Embodiment 1 of the present invention. FIG. 7 is a flowchart showing an outline of the composite display process performed by the image processing apparatus according to the first embodiment of the present invention. FIG. 8 is a flowchart showing an outline of processing performed by the image processing apparatus according to the first modification of the first embodiment of the present invention. FIG. 9 is a flowchart showing an outline of processing performed by the image processing apparatus according to the second modification of the first embodiment of the present invention. FIG. 10 is a flowchart showing an outline of processing performed by the image processing apparatus according to Embodiment 2 of the present invention. FIG. 11 is a flowchart showing an outline of a photographing process performed by the image processing apparatus according to the second embodiment of the present invention. FIG. 12 is a flowchart showing an overview of image processing performed by the image processing apparatus according to Embodiment 2 of the present invention. FIG. 13 is a diagram schematically illustrating an enlarged image generation process performed by the image processing apparatus according to the second embodiment of the present invention. FIG. 14 is a diagram showing an enlarged image group generated by the image processing apparatus according to Embodiment 2 of the present invention. FIG. 15 is a flowchart showing an overview of image processing performed by the image processing apparatus according to Embodiment 3 of the present invention. FIG. 16 is a diagram schematically showing an enlarged image generation process in the case where the quotient obtained by dividing the maximum value of the face height by the minimum value is larger than the reference enlargement ratio. FIG. 17 is a diagram illustrating an enlarged image group generated when the quotient obtained by dividing the maximum value of the face height by the minimum value is the reference enlargement ratio. FIG. 18 is a flowchart showing an outline of processing performed by the image processing apparatus according to Embodiment 4 of the present invention.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention (hereinafter referred to as “embodiments”) will be described with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a block diagram showing a configuration of an imaging apparatus which is an example of an image processing apparatus according to Embodiment 1 of the present invention. The imaging apparatus 1 shown in the figure mainly has a function of shooting a moving image and playing back the shot moving image and an image that has been subjected to image processing based on the moving image. Specifically, the imaging device 1 collects light from a predetermined region, and generates an image unit 2 that generates digital image data from an image signal obtained by photoelectrically converting the collected light, and imaging An image processing unit 3 that performs image processing on an image corresponding to the image data generated by the unit 2, a display unit 4 that displays various types of information including images processed by the image processing unit 3, and an imaging unit 2 The temporary storage unit 5 that temporarily stores the image data and the like generated by the camera, the storage unit 6 that stores various types of information including the image data of the image captured by the imaging unit 2, and the operation signal of the imaging device 1 including the release signal. The operation input unit 7 that receives inputs such as, a clock 8 having a timer function, and a control unit 9 that controls the operation of the imaging apparatus 1 are provided.

  The imaging unit 2 includes one or a plurality of lenses, and includes an optical system that collects light from a subject existing in a predetermined visual field area, a diaphragm that adjusts the amount of incident light collected by the optical system, and a release. An image sensor such as a CCD (Charge Coupled Device) that receives the light that has passed through the aperture and the shutter and converts it into an electrical signal, and an analog signal output from the image sensor. And a signal processing circuit that generates digital image data by performing A / D conversion after performing signal processing such as the above.

  The display unit 4 is realized by using a display panel made of liquid crystal, organic EL (Electro Luminescence), or the like, and appropriately displays operation information of the imaging apparatus 1 and information related to photographing in addition to image data.

  The image processing unit 3 includes a face detection unit 31 that detects the face of a person included in the image, a trimming unit 32 that performs trimming of the image, an image enlargement unit 33 that magnifies the image trimmed by the trimming unit 32, and a plurality of An image composition unit 34 for composing the images, and a compression / decompression unit 35 for compressing or decompressing the image data.

  The storage unit 6 includes an image data storage unit 61 that stores image data of an image captured by the imaging unit 2 and processed by the image processing unit 3, and a face detection unit 31 that detects a face in the image. It has a face information storage unit 62 that stores face information to be referenced and a program storage unit 63 that stores various programs executed by the imaging apparatus 1. The storage unit 6 is realized using a semiconductor memory such as a flash memory or a DRAM (Dynamic Random Access Memory) that is fixedly provided inside the imaging apparatus 1. Note that the storage unit 6 may function as a recording medium interface that reads information recorded on a recording medium while recording information on a computer-readable recording medium such as a memory card mounted from the outside. Good.

  The operation input unit 7 includes a power switch of the imaging apparatus 1, a release switch that inputs a release signal that gives an imaging instruction, a mode change switch that switches among various operation modes that can be set by the imaging apparatus 1, and the like. The operation input unit 7 may be realized by a touch panel stacked on the display unit 4 so that the user can input an operation signal based on information displayed on the display unit 4.

  The control unit 9 includes a trimming region determination unit 91 that determines a region to be trimmed in an image, a change detection unit 92 that detects a change in the shooting time based on a displacement amount of a feature point between adjacent images, and a series of plural A region extracting unit 93 that extracts a region that does not include the same face in the image and a display control unit 94 that controls the display of the display unit 4. The control unit 9 is realized using a CPU (Central Processing Unit) or the like, and is connected to each component of the imaging device 1 that is a control target via a bus line.

  The trimming unit 32, the image enlargement unit 33, the trimming region determination unit 91, the change detection unit 92, and the region extraction unit 93 are positions in the same face image detected by the face detection unit 31 for each frame constituting the moving image. And an image correction means for performing correction so that the sizes are uniform. Here, the face position is determined, for example, by the coordinates of one or more representative points of the face image, and the face size is determined, for example, as the area of the contour portion of the face image. Further, the fact that the positions of the faces are aligned means that the difference between some or all of the coordinates of one or more representative points in the images of the two faces is smaller than a predetermined range. Furthermore, the fact that the face sizes are equal means that the difference in the area of the contour portion of the face image between the two face images is smaller than a predetermined range.

Here, with reference to FIG. 2 to FIG. 5, characteristics of processing performed by the imaging apparatus 1 having the above configuration will be described. FIG. 2 is a diagram illustrating a captured image group as a continuous image group including a plurality of images continuously captured by the imaging apparatus 1. The moving image shown in FIG. 2 includes N photographed images (frames), and is arranged in the order of photographing along the arrow drawn at the bottom of the screen in FIG. In FIG. 2, among the N photographed images, seven photographed images F ₁ , F _i , F _{i + 1} , F _j , F _k , F _{k + 1} , F _N are shown (1 <I <j <k <N). Among these, the photographed images F _i and F _{i + 1} are adjacent to each other in photographing time. Also, the captured images F _k and F _{k + 1} are adjacent to each other.

  When the imaging apparatus 1 captures a series of moving images shown in FIG. 2, the face detection unit 31 detects the face of the same person in each captured image, and the trimming unit 32 determines the trimming region determined by the trimming region determination unit 91. Trimming is performed, and the image enlargement unit 33 enlarges the trimmed area, thereby generating an enlarged image as a corrected image.

FIG. 3 is a diagram showing the trimming area determined by the trimming area determination unit 91. In the case shown in FIG. 3, the trimming areas T ₁ , T _i , T _{i + 1} , T _j , T _k , T _{k + 1} , T _N are the captured images F ₁ , F _i , F _{i + 1} , F _j , These are regions determined for F _k , F _{k + 1} , and F _N , respectively, and are regions set so that the subject's face is the entire screen. In the case shown in FIG. 3, the aspect ratio of each trimming area is equal to the aspect ratio of the captured image.

FIG. 4 is a diagram illustrating an enlarged image group including enlarged images obtained by enlarging the images forming the captured image group by the image enlargement unit 33. The enlarged image group shown in the figure corresponds to a corrected image group corrected by the image correcting unit. FIG. 4 shows only a part of the enlarged images of the enlarged image group. The enlarged images E ₁ , E _i , E _{i + 1} , E _j , E _k , E _{k + 1} , E _N are trimmed by the image enlargement unit 33 by the trimming regions T ₁ , T _i , T _{i + 1} , T _j , T _k , T _{k + 1} , and T _N are enlarged images. In this way, by continuously displaying images composed of images obtained by enlarging the face portion of the subject, the user can enjoy changes in the facial expression of the subject in a specific scene. The enlarged images E ₁ , E _i , E _{i + 1} , E _j , E _k , E _{k + 1} , E _N are displayed on the display unit when the imaging device 1 is set to a facial expression reproduction mode which is a kind of reproduction mode. 4 is reproduced and displayed.

In the imaging apparatus 1, the composite display mode can be further set while the facial expression reproduction mode is set. FIG. 5 is a diagram illustrating a display example of an image group displayed on the display unit 4 when the imaging apparatus 1 is set to the composite display mode. The image group shown in the figure is a composite image group synthesized by the image synthesis unit 34. Note that FIG. 5 shows only a part of the composite images in the composite image group. The composite images C ₁ , C _i , C _{i + 1} , C _j , C _k , C _{k + 1} , C _N shown in FIG. 5 are taken images F ₁ , F _i , F _{i + 1} , F _j , F _k. , F _{k + 1} , F _N are enlarged images E ₁ , E _i , E _{i + 1} , E _j , E _k , E _{k + 1} , E _N which are arranged in a reduced size. When generating a composite image, first, the region extraction unit 93 extracts a region that does not include the face of a series of all the captured images F _{1 to} F _N. This extracted region is, for example, a region having the same aspect ratio as that of the captured image and having the largest image area. Following the processing by the region extraction unit 93, the image composition unit 34 generates a composite image by reducing and arranging the trimming region in the region extracted by the region extraction unit 93.

In the composite image group shown in FIG. 5, the emphasis icon m is superimposed and displayed on the enlarged image display areas of the composite images C _{i + 1} and C _{k + 1} . The emphasis icon m is displayed superimposed on a predetermined number of composite images from a composite image including a captured image in which the subject's facial expression changes greatly from the immediately preceding captured image. The design, display position, and size of the highlight icon m are not limited to those shown in FIG. Further, instead of displaying the emphasis icon m, for example, a voice may be output or the main body of the imaging apparatus 1 may be vibrated to notify that the facial expression change is large.

  In the above description, the case where the subject gradually approaches the imaging device 1 is captured with a moving image. However, similar image processing can be performed even when the subject moves away from the imaging device 1. Is possible.

  FIG. 6 is a flowchart illustrating an outline of processing performed by the imaging apparatus 1. First, when the imaging device 1 is set to the shooting mode (step S1: Yes), the display control unit 94 causes the display unit 4 to start displaying a through image (step S2).

  Thereafter, when the release switch of the operation input unit 7 is pressed (step S3: Yes), the imaging unit 2 takes an image (step S4). Subsequently, the face detection unit 31 detects a face included in the photographed image (step S5). At this time, the face detection unit 31 refers to the face information stored in the face information storage unit 62 and detects the face by a technique such as pattern matching.

  When the face detection unit 31 detects a face included in the image (step S6: Yes), the control unit 9 temporarily stores face information including the position and size of the face detected by the face detection unit 31 together with the captured image. It records in the part 5 (step S7). In step S7, when the face detection unit 31 detects a plurality of faces, for example, only the image closest to the center of the screen may be recorded in the temporary storage unit 5. When the face detection unit 31 detects a plurality of faces, only the face with the largest area may be recorded in the temporary storage unit 5.

  Subsequently, when the face image detected by the face detection unit 31 can be enlarged (step S8: Yes), the control unit 9 performs exposure and focus adjustment at the face position (step S9). Here, whether or not the face image can be enlarged is determined depending on whether or not the enlargement ratio when the face is enlarged to a predetermined size is equal to or less than a predetermined threshold value. The threshold value here is preferably a value that can clearly recognize the face when the image is enlarged at an enlargement ratio equal to or less than the threshold value.

  Thereafter, the trimming area determination unit 91 determines an area to be trimmed in the captured image (step S10). In this case, for example, as shown in FIG. 3, the trimming area is set as an area where the size of the face is almost the entire area of the screen. The trimming unit 32 performs image trimming according to the region determined by the trimming region determination unit 91 (step S11). Thereafter, the image enlargement unit 33 enlarges the trimming region to generate an enlarged image, and records the generated enlarged image in the temporary storage unit 5 (step S12).

  When there are two or more enlarged images temporarily stored in the temporary storage unit 5 (step S13: Yes), the change detection unit 92 compares the two most recent enlarged images and the facial expression of the subject common to both enlarged images. The change is detected, and the detection result is recorded in the temporary storage unit 5 (step S14).

  When the imaging end instruction is input from the operation input unit 7 while the imaging apparatus 1 performs the processes of steps S4 to S14 (step S15: Yes), the imaging apparatus 1 ends the imaging (step S16). On the other hand, when the end instruction is not input from the operation input unit 7 (step S15: No), the imaging device 1 returns to step S4 and repeats the process. The photographing end instruction in step S15 is input by pressing a release switch, for example.

  After the imaging device 1 finishes shooting, the control unit 9 reads the image data of the captured image and the enlarged image from the temporary storage unit 5, compresses the read image data by the compression / decompression unit 35, and then stores the storage unit 6. (Step S17). At this time, the control unit 9 also deletes the image data from the temporary storage unit 5. Thereafter, the imaging apparatus 1 returns to step S1.

  When the face detection unit 31 does not detect a face included in the image at step S6 (step S6: No), and when the face image detected by the face detection unit 31 is not expandable at step S8 (step S8). : No), the imaging apparatus 1 proceeds to step S13.

  In step S13, when there are no two or more enlarged images temporarily stored in the temporary storage unit 5 (step S13: No), the imaging device 1 proceeds to step S15.

  Next, a case where the imaging device 1 is not set to the shooting mode (step S1: No) will be described. In this case, when the imaging device 1 is set to the playback mode (step S21: Yes), the imaging device 1 performs processing according to the type of the playback mode that is set.

  When the imaging apparatus 1 is set to the facial expression reproduction mode (step S22: Yes) and is set to the composite display mode (step S23: Yes), the imaging apparatus 1 A composite image with the enlarged image is displayed in the order of photographing (step S24). More detailed processing contents of the composite display will be described later with reference to FIG. On the other hand, when the imaging device 1 is set to the facial expression reproduction mode (step S22: Yes) and is not set to the composite display mode (step S23: No), the imaging device 1 displays the enlarged image group (see FIG. 4). Is used to reproduce the moving image (step S25). If an end instruction is input from the operation input unit 7 after step S24 or S25 (step S26: Yes), the imaging device 1 ends a series of processes.

  In step S22, when the facial expression reproduction mode is not set (step S22: No), the imaging apparatus 1 performs normal moving image reproduction (step S27), and proceeds to step S26. The moving image reproduced in this case is a captured image group (see FIG. 2).

Next, the outline of the composite display process in step S24 will be described with reference to the flowchart shown in FIG. In FIG. 7, first, the region extraction unit 93 acquires the face positions of all the captured images F _{1 to} F _N from the face information storage unit 62, and based on the acquired position information, the face is detected in all the captured images. An area not included is extracted (step S31).

Subsequently, a loop process is performed on all the captured images F _{1 to} F _N (step S32). In this loop process, first, the image composition unit 34 acquires a pair of a captured image and a trimmed image corresponding to each other from the storage unit 6 via the compression / decompression unit 35 (step S33). Subsequently, the change detection unit 92 detects the presence or absence of a facial expression change based on, for example, the amount of displacement from the image before the representative point of the face (step S34). At this time, the first photographed image is set in advance so as to be treated as having no expression change.

In step S34, when the change detection unit 92 detects a change in facial expression (step S34: Yes), the image composition unit 34 generates a change emphasis composite image, and the display unit 4 displays the generated change emphasis composite image. (Step S35). The change-enhanced composite image here is a composite image that is displayed together with information for emphasizing the change at a location where the expression change is large. Specifically, the composite image C _{i + 1} , C _k shown in FIG. _{It is +1} . In the case of these composite images C _{i + 1} and C _{k + 1} , the emphasis icon m is nothing but information that emphasizes the change.

On the other hand, in step S34, when the change detection unit 92 does not detect a change in facial expression (step S34: No), the captured image to be processed is determined for a predetermined time (after the change detection unit 92 detects the expression change last). For example, when the image is taken after the elapse of time (step S36: Yes), the image composition unit 34 generates a normal composite image, and the display unit 4 displays the generated composite image. (Step S37). Composite image to be displayed in this step S37 is, for example, an image such as a composite image C _i shown in FIG.

  In step S36, when the image to be processed is an image taken before a predetermined time has elapsed since the change detection unit 92 last detected a facial expression change (step S36: No), the imaging apparatus 1 performs step S35. Migrate to Thereby, on the display unit 4, the emphasis icon m is continuously displayed for a predetermined number of times after the change detection unit 92 detects the expression change. Thus, by continuously displaying the highlight icon m, the user can surely recognize that the highlight icon m is displayed.

  When the loop process (steps S33 to S37) in step S32 described above is executed for all captured images, the imaging apparatus 1 returns to the main routine.

  According to Embodiment 1 of the present invention described above, the same face is detected by detecting the same face included in each of a plurality of images constituting a continuous image group (captured image group) that has been continuously captured. Each image is corrected so that the size and position of each image are aligned, a corrected image is generated, and a corrected image group or a continuous image group including the generated corrected image is displayed. Therefore, the user can enjoy various state changes of the subject using a plurality of images taken continuously.

  Further, according to the first embodiment, since the enlarged image of the face is combined with the captured image and displayed, the facial expression change and the body movement can be displayed together.

  Further, according to the first embodiment, when displaying a composite image, when the facial expression change is large, the emphasis icon is superimposed and displayed, so that the facial expression change is visually emphasized. can do.

(Modification 1)
FIG. 8 is a flowchart illustrating an outline of processing performed by the imaging apparatus according to the first modification of the first embodiment. The configuration of the imaging apparatus according to the first modification is the same as the configuration of the imaging apparatus 1. 8 is different from the flowchart of FIG. 6 in that a process is added between step S14 and step S15. Thereafter, in step S14, the change detection unit 92 compares the two most recent enlarged images, detects a facial expression change of the subject common to both enlarged images, and records the detection result in the temporary storage unit 5 to continue processing. (Steps S41 to S43) will be described.

  When a change in facial expression is detected by the change detection unit 92 (step S41: Yes), the image composition unit 34 generates a change-emphasized composite image, and the display unit 4 displays the generated change-emphasized composite image (step S42). ). Thereafter, the imaging apparatus 1 proceeds to step S15.

  On the other hand, when the facial expression change is not detected by the change detection unit 92 (step S41: No), the image to be processed is processed for a predetermined time (for example, after the facial expression change is detected by the change detection unit 92 last time). When the image is taken after the elapse of time (about 0.5 to 2 seconds) (step S43: Yes), the image composition unit 34 generates a normal composite image, and the display unit 4 displays the generated composite image. (Step S44). Thereafter, the imaging apparatus 1 proceeds to step S15.

  In step S43, when the image to be processed is an image taken before a predetermined time has elapsed since the change detection unit 92 detected the last facial expression change (step S43: No), the imaging apparatus 1 performs step S42. Migrate to

  According to the first modification of the first embodiment described above, the photographer can take a picture while watching a moving image based on the composite image in almost real time.

(Modification 2)
FIG. 9 is a flowchart showing a flow of control of the imaging apparatus according to the second modification of the first embodiment. The configuration of the imaging apparatus according to the second modification is the same as that of the imaging apparatus 1. 9 is different from the flowchart of FIG. 6 in that the imaging apparatus 1 finishes shooting (step S16), reads the image data of the captured image and the enlarged image from the temporary storage unit 5, and stores them through the compression / decompression unit 35. This is the point of performing the same composite display (step S45) as step S24 in FIG. 6 after storing in the unit 6 (step S17). According to the second modification of the first embodiment as described above, the photographer can enjoy the change in the expression of the subject by watching the moving image by the composite image of the image photographed immediately after the photographing is finished.

(Embodiment 2)
The second embodiment of the present invention is characterized in that other images are enlarged in accordance with the image having the largest face size of the subject in the series of images. The imaging apparatus according to the second embodiment has the same configuration as that of the imaging apparatus 1 described in the first embodiment.

  FIG. 10 is a flowchart illustrating an outline of processing performed by the imaging apparatus 1 according to the second embodiment. First, when the imaging device 1 is set to the shooting mode (step S51: Yes), the display control unit 94 causes the display unit 4 to start displaying a through image (step S52).

  Thereafter, when the release switch of the operation input unit 7 is pressed (step S53: Yes), the imaging apparatus 1 performs an image capturing process (step S54). FIG. 11 is a flowchart showing an outline of the photographing process in step S54. In FIG. 11, first, the imaging unit 2 captures an image (step S71). Thereafter, the face detection unit 31 refers to the face information stored in the face information storage unit 62 to detect a face included in the image captured by the imaging unit 2 (step S72).

  When the face detection unit 31 detects a face included in the image (step S73: Yes), the control unit 9 temporarily stores face information including the position and size of the face detected by the face detection unit 31 together with the captured image. It records in the part 5 (step S74).

  Subsequently, when the size of the face detected by the face detection unit 31 is equal to or larger than a predetermined size (step S75: Yes), a comparison reference image that is a comparison target of the captured image is temporarily stored in the temporary storage unit 5. If so (step S76: Yes), the change detection unit 92 compares the captured image with the comparison reference image (step S77). As a result of the comparison, when there is a change in the position and size of the face that exceeds a predetermined standard (step S78: Yes), the control unit 9 stores the captured image as a processing image (step S79), and is shown in FIG. Return to the main routine. At this time, the image data of the captured image added with identification information indicating that it is a processing image is recorded in the image data storage unit 61.

In step S76, when the comparison reference image is not stored in the temporary storage unit 5 (step S76: No), the control unit 9 records the captured image as the comparison reference image in the temporary storage unit 5 (step S80). Return to the main routine. For example, in the photographed image group shown in FIG. 2, the photographed image F ₁ is a comparison reference image. In this case, the captured images F _i , F _{i + 1} , F _j , F _{j + 1} , F _k , F _{k + 1} , and F _N also have different face positions and sizes from the comparison reference image. It is stored in the image data storage unit 61.

  In step S75, when the face size is smaller than the predetermined size (step S75: No), if the comparison reference image is temporarily stored in the temporary storage unit 5 (step S81: Yes), the control unit 9 performs the comparison. The reference image is deleted (step S82), and the process returns to the main routine shown in FIG.

  When the face size is smaller than the predetermined size (step S75: No), and the comparison reference image is not temporarily stored in the temporary storage unit 5 (step S81: No), the control unit 9 returns to the main routine.

  When an imaging end instruction is input by the operation input unit 7 while the imaging apparatus 1 is performing the imaging process of Step S54 (Steps S71 to S82) (Step S55: Yes), the imaging apparatus 1 ends the imaging. Then, a predetermined image (processing image) is processed (step S57). On the other hand, when the imaging input instruction is not input by the operation input unit 7 while the imaging apparatus 1 is performing the imaging process of step S54 (step S55: No), the imaging apparatus 1 repeats the imaging process of step S54.

FIG. 12 is a flowchart showing an overview of the image processing process in step S57. First, when there are a predetermined number or more of processing images stored in the image data storage unit 61 (step S91: Yes), a processing image including a face image having the maximum area is extracted and the maximum processing image I _{max is obtained.} and then (step S92), the height of the face of the maximum processing image I _max (length of the vertical) and f _max (step S93).

Thereafter, the imaging device 1 performs a loop process on all the processing images F _n (step S94). Control unit 9 first magnification height f _n of the face in the processing image F _n K _(n) = calculates the f _max / f _n (step S95). Thereafter, the image enlargement unit 33 aligns the face position of the processing image F _{n with} the face position of the maximum processing image I _max and enlarges the processing image F _n at the enlargement ratio K (n) (step S96).

  Subsequently, the trimming area determination unit 91 determines a trimming area (step S97). Thereafter, the trimming unit 32 performs image trimming according to the trimming region determined by the trimming region determination unit 91 (step S98). Thereafter, the control unit 9 records the trimmed enlarged image in the storage unit 6 (step S99).

  After the loop process (steps S95 to S99) in step S94 is completed, the imaging apparatus 1 returns to the main routine shown in FIG.

  A case where the number of processing images is smaller than the predetermined number in step S91 (step S91: No) will be described. In this case, when the facial expression change detected by the change detection unit 92 is large (step S100: Yes), the image processing unit 3 performs a slow motion response process on the processing image (step S101). The slow motion handling process in step S101 is, for example, a process that makes the display time of each processing image longer than usual. Thereafter, the imaging apparatus 1 returns to the main routine shown in FIG. In step S100, when the change in facial expression is small (step S100: No), the imaging apparatus 1 returns to the main routine shown in FIG.

FIG. 13 is a diagram schematically illustrating the generation of an enlarged image in the loop process of step S94. In FIG. 13, the enlarged image E ′ ₁ of the processing image F ₁ is enlarged at an enlargement factor K (1) = f _max / f ₁ . As a result, the height of the face in the enlarged image E ′ ₁ is f _max .

FIG. 14 is a diagram illustrating the generated enlarged image group. In FIG. 14, the enlarged images E ′ ₁ , E ′ _i , E ′ _{i + 1} , E ′ _j , E ′ _k , E ′ _{k + 1} , E ′ _N are processed images F ₁ , F _i , F _{i. The} images are enlarged after trimming ₊₁ , F _j , F _k , F _{k + 1} , and F _N , respectively. The position and size of the face in these enlarged images are displayed so as to be substantially aligned with the position and size of the face of the maximum processing image I _max . Accordingly, by displaying the enlarged image group shown in FIG. 14 in the order of photographing, the user can recognize the change in the facial expression of the subject. As described above, in the second embodiment, only the image having a large change in the position and size of the face is extracted from the comparative reference image having a certain size, and the extracted photographed image is used as the processing image. The load applied to the imaging apparatus 1 when performing image processing can be reduced.

The processes in steps S61 to S67 subsequent to the case where the imaging device 1 is not set to the shooting mode (step S51: No) sequentially correspond to the processes in steps S21 to S27 shown in FIG. Note that the composite image to be displayed in the composite display in the step S64 is in the selected captured image F _n as processing an image, the enlarged image E _n 'being enlarged by the captured image F _n a predetermined magnification rate F (n) It is an image obtained by combining.

  According to the second embodiment of the present invention described above, as in the first embodiment, the user can enjoy various state changes of the subject using a plurality of images taken continuously.

  Further, according to the second embodiment, only the image having a large change in the position and size of the face is extracted from the comparative reference image that is somewhat large, and only the extracted photographed image is used as the processing image. Thus, it is possible to reduce the load applied to the imaging apparatus when performing image processing, and to realize high-speed and stable processing.

  Further, according to the second embodiment, since the face is enlarged in accordance with the maximum processing image having the largest face size, when the body is also displayed in the maximum processing image, the user You can enjoy the change of movement at the same time.

(Embodiment 3)
In the third embodiment of the present invention, as in the second embodiment described above, another image is enlarged in accordance with the image having the largest face size of the subject in the series of images. When there is a possibility that the resolution of the image is lowered due to being too large, the enlargement ratio is reduced to suppress the decrease in the resolution of the image. The imaging apparatus according to the third embodiment has the same configuration as that of the imaging apparatus 1 described in the first embodiment.

The outline of the process performed by the imaging apparatus 1 according to the third embodiment is the same as the outline of the process described with reference to the flowchart shown in FIG. 10 in the second embodiment, except for the image processing process. Therefore, hereinafter, an outline of the image processing performed by the imaging apparatus 1 according to the third embodiment will be described with reference to a flowchart shown in FIG. First, when there are a predetermined number or more of processing images stored in the image data storage unit 61 (step S111: Yes), the control unit 9 extracts a processing image including a face image having the largest area and extracts the maximum processing image. The processing image I _max is set (step S112), and the face height of the maximum processing image I _max is set to f _max (step S113). Subsequently, the processing image including the face image having the smallest area is extracted and set as the minimum processing image I _min (step S114), and the face height of the minimum processing image I _min is set as f _min (step S114). S115). In the case illustrated in FIG. 2, the maximum processing image I _max is the captured image F _N , and the minimum processing image I _min is the captured image F ₁ .

Thereafter, the imaging device 1 performs a loop process on all the processing images F _n (step S116). First, when the maximum processing image I _max quotient obtained by dividing the height f _max of the face at the height f _min of the face in the minimum feature image I _min at f _max / f _min is greater than the predetermined reference magnification M ( step S117: Yes), the control unit 9, the magnification of the height f _n of the face in the processing image _{F n K (n) = f} max / f n , the value alpha × K multiplied by the constant smaller than 1 alpha Let (n) be a true enlargement ratio (step S118). Here, the reference enlargement ratio M is a value determined according to the resolution of the image, for example, M = 5. Further, the constant α can be set to α = 0.5, for example.

In step S117, when the quotient f _max / f _min is equal to or smaller than the reference enlargement ratio M (step S117: No), the control unit 9 calculates f _max / f _n as the enlargement ratio K (n) (step S119).

After step S118 or S119, the image enlarging unit 33 aligns the face position of the processing image F _{n with} the face position of the maximum processing image I _max and enlarges it with the enlargement factor K (n) (step S120). Thereafter, the trimming area determination unit 91 determines an area to be trimmed with the processing image F _n (step S121). Subsequently, the trimming unit 32 performs image trimming according to the trimming region determined by the trimming region determination unit 91 (step S122). Thereafter, the control unit 9 records the trimmed enlarged image in the storage unit 6 (step S123).

FIG. 16 is a diagram schematically illustrating generation of an enlarged image when f _max / f _min > M in the loop processing of step S116. 16, the enlarged image E '' ₁ of the processing image F ₁ is the minimum processing image I _min is enlarged by the enlargement ratio K (1) = α × f max / f 1. As a result, the height of the face in the enlarged image E ″ ₁ is α × f _max (<f _max ).

FIG. 17 is a diagram illustrating an enlarged image group generated when f _max / f _min > M. In FIG. 17, the enlarged images E ″ ₁ , E ″ _i , E ″ _{i + 1} , E ″ _j , E ″ _k , E ″ _{k + 1} , E ″ _N are shown in FIG. The captured images F ₁ , F _i , F _{i + 1} , F _j , F _k , F _{k + 1} , and F _N are enlarged images after trimming. These enlarged image, the height of the vertical length and the face of the screen is not large enough processing image F _L, is enlarged enough to identify facial expressions. Therefore, the user can fully recognize facial expression changes.

  The processing in steps S124 and S125 performed after the processing image is smaller than the number corresponding to the moving image of the predetermined time in step S111 (step S111: No) sequentially corresponds to the processing in steps S100 and S101 in FIG. ing.

  According to the third embodiment of the present invention described above, as in the first embodiment, the user can enjoy various state changes of the subject using a plurality of images taken continuously.

  Further, according to the third embodiment, when there is a possibility that the enlargement ratio is too large and the resolution is lowered, priority is given to the resolution by reducing the enlargement ratio, thereby clarifying facial expressions and body movements. Can be displayed.

(Embodiment 4)
FIG. 18 is a flowchart showing an outline of processing performed by the imaging apparatus according to Embodiment 4 of the present invention. In the fourth embodiment, the process of enlarging and trimming the face image is performed when the playback mode is executed. The imaging device according to the fourth embodiment has the same configuration as the imaging device 1 described in the first embodiment.

  When the imaging device 1 is set to the shooting mode (step S131: Yes), the processing of steps S132 to S136 sequentially corresponds to the processing of steps S52 to S56 of FIG. The imaging apparatus 1 returns to step S131 after shooting is completed (step S136).

Next, a case where the imaging device 1 is not set to the shooting mode (step S131: No) will be described. In this case, when the imaging device 1 is set to the playback mode (step S137: Yes) and the facial expression playback mode is set (step S138: Yes), the control unit 9 is included in the captured image group to be played back. For the processed image to be processed, face determination relating to the face to be processed is performed based on information stored in the image data storage unit 61 (step S139). The face determination here means determination of the time during which the face to be processed is continuously photographed, and determination of the change in the position and size of the face detected by the change detection unit 92. As a result of the face determination, when the same face is continuously photographed for a predetermined time (for example, 2 seconds) or longer (step S140: Yes) and the change in the position and size of the face is large (step S141: Yes), control is performed. The unit 9 sets the processing image including the face image having the maximum area in the series of images as the maximum processing image I _max (step S142), and sets the face height in the maximum processing image I _max as f _max . (Step S143).

Subsequently, a loop process is performed on all the processing images F _n to be reproduced (step S144). The loop processing (steps S145 to S149) performed here is substantially the same as the loop processing (steps S95 to S99) of step S94 in FIG.

  After the loop process in step S144 is completed, the display control unit 94 causes the display unit 4 to play back the enlarged image movie (step S150). This moving image is obtained by continuously displaying a series of enlarged image groups as shown in FIG. 14, for example.

  Thereafter, when an end instruction is input by the operation input unit 7 (step S151: Yes), the imaging device 1 ends a series of processes. On the other hand, when the end instruction is not input by the operation input unit 7 (step S151: No), the imaging apparatus 1 returns to step S131.

  In step S137, when the playback mode is not set (step S137: No), the imaging device 1 proceeds to step S150.

  In step S138, when the imaging device 1 is not set to the expression playback mode (step S138: No), the imaging device 1 performs normal playback (step S153), and proceeds to step S150.

  When the same face is not continuously photographed for a predetermined time or longer in step S140 (step S140: No), and when the change in the position or size of the face is not larger than the reference in step S141 (step S141: No), The display control unit 94 causes the display unit 4 to display a message indicating that reproduction with emphasis on the facial expression to be performed in the facial expression reproduction mode is not possible (step S152). Thereafter, the imaging apparatus 1 proceeds to step S150.

  According to the fourth embodiment of the present invention described above, as in the first embodiment, the user can enjoy various state changes of the subject by using a plurality of images taken continuously.

  In the fourth embodiment, instead of the loop process in step S144, a loop process similar to the loop process in step S116 of FIG. 15 described in the third embodiment may be performed.

  Up to this point, the mode for carrying out the present invention has been described, but the present invention should not be limited by the above-described four embodiments. For example, according to the present invention, it is possible to perform the same processing as described above using still images taken continuously.

  Further, in the present invention, when a corrected image is generated by enlarging and trimming an image at the time of image reproduction as in the fourth embodiment, at least the processing image to be reproduced may be stored in the storage unit. . Therefore, in this case, the present invention can also be realized by an image processing apparatus having a configuration in which the imaging unit 2 is removed from the imaging apparatus 1 shown in FIG.

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Imaging part 3 Image processing part 4 Display part 5 Temporary storage part 6 Storage part 7 Operation input part 8 Clock 9 Control part 31 Face detection part 32 Trimming part 33 Image expansion part 34 Image composition part 35 Compression / decompression part 61 Image Data storage unit 62 Face information storage unit 63 Program storage unit 91 Trimming region determination unit 92 Change detection unit 93 Region extraction unit 94 Display control unit m Icon for emphasis

Claims (12)

A storage unit that stores information related to each of the plurality of images and a continuous image group composed of a plurality of images photographed continuously;
A display unit capable of continuously displaying the continuous image group stored in the storage unit in a shooting order;
A face detection unit for detecting the same face included in each of the plurality of images stored in the storage unit;
Image correcting means for correcting each image so that the position and size in each image of the same face detected by the face detection unit are aligned;
A display control unit that performs control for causing the display unit to display one of the corrected image group including the corrected image of each image corrected by the image correcting unit and the continuous image group;
An image processing apparatus comprising: The image correcting means includes
The comparison reference image is determined from the plurality of images, and the image is corrected with respect to an image in which the size and position of the same face are changed as compared with the comparison reference image. Image processing apparatus. The image correcting means includes
A trimming area determining unit that determines a trimming area in each of the plurality of images;
A trimming unit that trims each image according to the trimming region determined by the trimming region determination unit;
An image enlarging unit that generates the corrected image by enlarging the image trimmed by the trimming unit to the same size as the plurality of images;
The image processing apparatus according to claim 1, further comprising: The image correcting means includes
An image enlargement unit for enlarging each image by an enlargement ratio given by a quotient obtained by dividing the maximum value of the length of the same face in a predetermined direction by the length in each image;
A trimming region determination unit for determining a trimming region in the image enlarged by the image enlargement unit;
A trimming unit that generates the corrected image by trimming each image according to the trimming region determined by the trimming region determination unit;
The image processing apparatus according to claim 1, further comprising: The image enlargement unit
When the quotient obtained by dividing the maximum value of the length of the same face in the predetermined direction by the minimum value of the length is larger than the standard enlargement factor, a product obtained by multiplying the enlargement factor by a constant smaller than 1 is set as the true enlargement factor. The image processing apparatus according to claim 4, wherein each image is enlarged.   The image processing apparatus according to claim 3, wherein an aspect ratio of the trimming region is equal to an aspect ratio of the plurality of images.   7. An image composition unit for generating a composite image by reducing the correction image and disposing the reduced correction image in a partial area of the image before correction. The image processing apparatus according to any one of the above. The image correcting means includes
A region extraction unit that extracts a region that is common to all of the plurality of images and does not include the same face;
The image composition unit
The image processing apparatus according to claim 7, wherein the corrected image corresponding to each image is arranged in a region extracted by the region extraction unit of each image. The image correcting means includes
A change detecting unit that detects a change in facial expression of the same face between two images of which the shooting times are adjacent among the plurality of images based on a displacement amount of the feature point of the same face;
The image composition unit
When the change in facial expression of the same face detected by the change detection unit is larger than a reference, an image indicating that the change in facial expression is large is combined with an image captured later of the two images. The image processing apparatus according to claim 7 or 8. The display control unit
Control that causes the display unit to display any one image group among three image groups obtained by adding a composite image group composed of composite images of the images synthesized by the image composition unit to the corrected image group and the continuous image group. The image processing apparatus according to claim 7, wherein: An imaging unit that collects light from a predetermined area and images a subject in the area and generates electronic data of the captured image,
The storage unit
Storing a plurality of captured images continuously captured by the imaging unit;
The display control unit
11. The control according to claim 10, wherein after the imaging unit finishes capturing the plurality of captured images, control is performed to display any one of the three image groups on the display unit. Image processing device. An imaging unit that collects light from a predetermined area and images a subject in the area and generates electronic data of the captured image,
The storage unit
Storing a plurality of captured images continuously captured by the imaging unit;
The display control unit
While the imaging unit is capturing the plurality of captured images, the display unit is controlled to sequentially display the correction image or the composite image corresponding to the latest captured image. The image processing apparatus according to any one of claims 7 to 9.

Images