JP2011234269A - Imaging device and image data generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device capable of obtaining various moving images, and to provide an image data generation method.SOLUTION: An imaging device comprises: an imaging unit 2 which images a subject and outputs image data of a moving image; a motion analyzing unit 3ca which analyzes the motion of the subject on the basis of image data 31a to 31e; an effect applying unit 3b which performs effect image processing of the image data according to the motion of the subject analyzed by the motion analyzing unit 3ca; and a recording unit 6 which records image data 33b to 33e subjected to effect image processing by the effect applying unit 3b.

Description

  The present invention relates to an imaging device and an image data generation method, and more particularly to an imaging device and an image data generation method capable of automatically selecting image processing at the time of imaging.

  In recent years, portable devices (photographing devices) with portable functions such as digital cameras have expanded the photographing area by using image processing. In addition, various image processing has become available to users, and it has become possible to easily enjoy making pictures reflecting the sensitivity of the photographer.

  For example, Patent Document 1 discloses a video recording apparatus in which animation data and character string data are read out by operating a digital effect key, and an image with a special effect can be enjoyed. Further, Patent Document 2 discloses an image composition device that performs a visual action according to the motion of a moving image.

JP-A-2005-347886 Japanese Patent No. 4142427

  As described above, various proposals have been made for processing an image. However, the video recording apparatus disclosed in Patent Document 1 only applies a special effect to a still image, and does not consider any special effect to a moving image. The image synthesizing apparatus disclosed in Patent Document 2 is to apply a special effect to a moving image. However, it only reacts to the movement of the moment, it does not analyze the process of movement and adjust the special effects on the whole movement. In the case of a moving image, the movement is important. If the image processing is changed in accordance with the entire movement, an image rich in change is obtained. However, a photographing apparatus of such a viewpoint has not been proposed conventionally.

  The present invention has been made in view of such circumstances, and provides a photographing apparatus and an image data generation method capable of acquiring a moving image rich in change without performing a special operation during photographing. The purpose is to do.

  In order to achieve the above object, a photographing apparatus according to a first aspect of the present invention includes an imaging unit that captures a subject image and outputs image data, and a change in the orientation of a face portion in a person image based on the image data. A face change determination unit that determines a start time and an end time of the image, and an image processing unit that performs image processing of the image data based on the start time and the end time determined by the face change determination unit. To do.

  In the photographing apparatus according to a second aspect of the present invention, in the first aspect, the face change determination unit further determines a change in the size of the face portion, and the image processing unit determines the size of the face. The content of the image processing is determined based on the change.

  According to a third aspect of the present invention, there is provided an imaging device that captures a subject image and outputs moving image data, a motion analysis unit that analyzes the motion of the subject based on the image data, and the motion analysis unit. The image processing apparatus includes an effect applying unit that performs an effect image process on the image data according to the analyzed movement of the subject, and a recording unit that records the image data subjected to the effect image process by the effect applying unit. .

According to a fourth aspect of the present invention, there is provided the photographing apparatus according to the third aspect, further comprising: determining an area on which the effect applying unit performs the effect image processing based on the movement of the subject analyzed by the motion analyzing unit. It has a start point / end point determination unit.
According to a fifth aspect of the present invention, in the fourth aspect, the effect applying unit performs the effect image processing on the image data between the start point and the end point determined by the operation start point / end point determination unit. Apply.

According to a sixth aspect of the present invention, in the photographing apparatus according to the third aspect, the motion analysis unit classifies the movement of the subject into a predetermined pattern.
In the photographing apparatus according to a seventh aspect based on the sixth aspect, the effect imparting unit changes the effect image processing according to the pattern by the motion analysis unit.
In the photographing apparatus according to the eighth aspect based on the third aspect, the motion analysis unit analyzes the movement of the subject using at least one of the direction and size of the face of the person of the subject.

  According to a ninth aspect of the present invention, there is provided an image data generation method for capturing a subject image and outputting moving image data, analyzing the motion of the subject based on the image data, and according to the analyzed subject motion. Then, effect image processing is performed on the image data, and the image data subjected to the effect image processing is recorded.

  According to the present invention, since image processing is performed in accordance with the movement of a subject, a photographing apparatus and an image data generation method capable of acquiring a moving image rich in change without performing a special operation during photographing. Can be provided.

1 is a block diagram mainly showing an electrical configuration of a camera according to an embodiment of the present invention. It is a block diagram which shows the detail of the image processing part of the camera concerning one Embodiment of this invention. It is a figure which shows a mode that the to-be-photographed object is image | photographed in the camera concerning one Embodiment of this invention. 4A and 4B are diagrams for explaining image processing when a moving subject is photographed in the camera according to the embodiment of the present invention. FIG. 5A is an image before image processing is performed, and FIG. It is the image after giving. It is a figure which shows a mode that the to-be-photographed object is image | photographed in the camera concerning one Embodiment of this invention. 4A and 4B are diagrams for explaining image processing when a subject with another motion is photographed in the camera according to the embodiment of the present invention, FIG. 5A is an image before image processing is performed, and FIG. It is an image after processing. In the camera according to an embodiment of the present invention, it is a diagram for explaining the detection of the movement of the face portion of the person of the subject, (a) when the face is facing the front, (b) is the right side (C) shows the case where the face is facing the left side. In the camera concerning one embodiment of the present invention, it is a figure explaining the case where the motion of the face part of the person of a subject is detected by tracking the shadow of a face, and (a)-(c) is a face from the front. The state of gradually moving toward the right side is shown. In the camera according to an embodiment of the present invention, it is a diagram showing the movement of the face portion of the subject person, (a) shows the change in the distance from the center of the shadow of the eyes and nose, (b) is the figure of the person It is a graph which shows the change of the size of a head. It is a flowchart which shows the operation | movement of the camera control of the camera concerning one Embodiment of this invention. It is a flowchart which shows the operation | movement of the locus | trajectory determination of the camera concerning one Embodiment of this invention. In the camera concerning one embodiment of the present invention, it is a figure showing the motion of the face part of the person of a subject, and (a)-(c) is a figure showing change when a person's head goes up and down. . It is a flowchart which shows the operation | movement of the starting point end point determination of the camera concerning one Embodiment of this invention. It is a flowchart which shows the operation | movement of the effect image process of the camera concerning one Embodiment of this invention.

  Hereinafter, a preferred embodiment will be described using a camera to which the present invention is applied according to the drawings. FIG. 1 is a block diagram showing an electrical configuration of a camera 10 according to an embodiment of the present invention. The camera 10 is a digital camera, and includes a control unit 1, an imaging unit 2, a face detection unit 2a, an image processing unit 3, a recording unit 6, a clock unit 7, a display unit 8, an operation unit 9, and the like.

  The imaging unit 2 includes a photographing lens (zoom lens) having a zoom function, an exposure control unit such as a shutter / aperture, an imaging element, a driving and reading circuit for the imaging element, and the like, and captures an object image formed by the photographing lens. It is converted into image data by the element and output. In this specification, a signal based on an image signal output from the image sensor is referred to as image data including an image processed signal.

  The face detection unit 2a is output from the imaging unit 2, and based on the image data processed by the image processing unit 3 in the control unit 1, whether or not a human face is included in the subject image, If it is included, its position and size are detected, and the detection result is output to the control unit 1. In detecting a face, shadows of parts such as eyes, nose, and mouth in the face are extracted and performed by a pattern matching method or the like. If the face can be detected, the size of the head (face size) is also detected.

  The control unit 1 controls the entire sequence of the camera 10 according to a stored program. The control unit 1 includes a recording control unit 1a, a face change determination unit 1b, a mode setting unit 1c, and an image processing unit 3. The recording control unit 1a controls recording of still image or moving image image data in the recording unit 6. In this control, when the moving image art mode is set, the image is analyzed, and control is performed so as to be recorded after performing image processing on a predetermined operation section.

  When the face detection unit 2a detects a face portion of a person, the face change determination unit 1b determines a change in the detected face orientation. The mode setting unit 1c is a processing unit for setting a moving image art mode on a menu screen or the like. As will be described later, in the moving image art mode, when the orientation of the person's face is within a predetermined range using the determination result of the face change determination unit 1b, image processing is performed within the range and the change is rich. This is a mode for creating a moving image.

  The image processing unit 3 in the control unit 1 includes a temporary recording unit 3a, an effect applying unit 3b, and other image processing units. The other image processing unit captures the image data output from the imaging unit 2, performs various image processing such as thinning processing, clipping processing, edge enhancement, color correction, image compression, and the like for live view display and recording of moving images. The image processing such as recording of still images and moving images and reproduction / display of still images and moving images is performed in the unit 6. In image compression / expansion, a still image compression / expansion circuit, a moving image compression / expansion circuit, and the like are provided for still images and moving images in accordance with respective processes.

  As will be described later, the temporary recording unit 3a is a memory for temporarily recording moving image data. The effect applying unit 3b is a temporarily recorded moving image when the moving image art mode is set. Image processing that gives an image effect to the image is performed. Details of the image processing unit 3 will be described later with reference to FIG.

  The operation unit 9 includes various operation members such as a release button, a power switch, a reproduction mode setting button, a menu button, etc., determines the operation state of these operation members, and sends a determination result to the control unit 1. The above-described control unit 1 controls photographing and reproduction in a predetermined sequence according to the operation state of the operation member. The movie art mode is set on the menu screen as described above, but in addition to this, a movie art mode button for setting the movie art mode is provided, and the movie art mode may be set by operating this button. I do not care. In addition, the art mode may be set at the time of moving image shooting.

  The recording unit 6 records image data of still images and moving images acquired by the imaging unit 2 and compressed by the image processing unit 3 when a shooting instruction is given by the release button. The display unit 8 includes a display unit such as a liquid crystal panel or an organic EL panel arranged on the back surface of the camera 10, and displays live view display before shooting, normal playback display of recorded images, camera information, and menus. Display the screen. The photographer can observe the live view display and determine the composition and timing. In addition, an image shot in the moving image art mode at the time of moving image shooting is also displayed.

  The clock unit 7 performs a time measuring operation and outputs photographing date / time information. This shooting date / time information is recorded when image data of a still image or a moving image is recorded in the recording unit 6. In addition, when shooting in the moving image art mode, it is also used for a timing operation in moving image shooting.

  Next, details of the configuration of the image processing unit 3 will be described with reference to FIG. As described above, the image processing unit 3 takes in the image data output from the imaging unit 2 and performs various image processing such as thinning processing and clipping processing. FIG. 2 illustrates image data generation in the moving image art mode. A related configuration is shown and this configuration will be mainly described.

  The temporary recording unit 3a includes memories 0 to n for temporarily recording the image data from the imaging unit 2 and the face detection result from the face detection unit 2a. The output of the face detection result temporarily recorded in the memories 0 to n is connected to the motion analysis unit 3ca. The motion analysis unit 3ca analyzes changes in the face trajectory and size.

  The motion analysis unit 3ca is connected to the motion start point / end point determination unit 3cb, and the motion start point / end point determination unit 3cb uses the analysis result of the motion analysis unit 3ca to determine the start point and end point of the predetermined motion of the person. As the predetermined operation, for example, there is a case where a person turns while dancing, and the starting point in this case corresponds to an arbitrary position and a position that goes around from that position. In addition to this turning action, there are various predetermined actions such as a turning action and an action of turning backward and leaving far away.

  The output of the operation start point / end point determination unit 3cb and the output of moving image data temporarily recorded in the temporary recording unit 3a are connected to an adoption determining unit 3ba which is one of the effect applying units 3b. The adoption deciding unit 3ba decides to adopt the image data determined by the operation start point / end point determination unit 3cb to be between the start point and the end point. When outputting the image data from the adoption determining unit 3ba, both the image data decided to be adopted and the image data not decided to be adopted are outputted. At this time, the image data decided to be adopted or the image data not decided to be adopted is used. Is output so that it can be identified.

  The output of the adoption determining unit 3ba is connected to an effect applying unit 3bb, which is one of the effect applying units 3b. The effect applying unit 3bb provides a predetermined image for the image data determined to be adopted by the employment determining unit 3ba. Perform image processing to add effects. As the predetermined image effect, for example, image processing such as changing the color or pattern of a person's clothes, changing the color tone to a sepia tone, making it gradually brighter, or making it darker is performed. For image data that has not been determined to be adopted, the image data is output as it is without giving any effect.

  The output of the effect applying unit 3bb is connected to the moving image compressing unit 3d, and the moving image compressing unit 3d performs image compression of the moving image data output from the effect applying unit 3bb. The moving image compression unit 3 d is connected to the recording unit 6, and the compressed image data is recorded in the recording unit 6.

  As described above, since the image processing unit 3 in the present embodiment is configured, as shown in FIG. 5, whether or not the person is turned away by the motion analysis unit 3 ca as shown in FIG. 3 described later. It is possible to analyze a predetermined operation such as rotating when a person turns around. Based on the analysis result, the motion start point / end point determination unit 3cb determines the start point and the end point in one rotation when the person is rotating, and when looking back, the other side looks back. It is possible to determine the start point and end point of the movement until the size of the head when viewing is reduced to half. When the start point and the end point are determined, the effect imparting unit 3b imparts an image effect to the image data in the meantime and records it in the recording unit 6. In the present embodiment, the internal configuration of the image processing unit 3 is a hardware configuration, but may be appropriately realized by software.

  Next, the application of image effects in the present embodiment will be described with reference to FIGS. FIG. 3 shows a scene in which the person 21 who was looking at the photographer side leaves toward the opposite side. This is a familiar scene that appears in movies and TV dramas, and is commonly used as a departure or parting expression. In such a scene, emotion can be enhanced by opening and ending a moving image with a story, and by adding a special effect, the image can be enhanced and the emotion can be amplified.

  Therefore, in the present embodiment, a moving image of the person 21 is acquired by the camera 10. As a result, the image data corresponding to the images 31a to 31e is temporarily recorded in the temporary recording unit 3a in the camera 10 as shown in FIG. By using the motion analysis unit 3ca and the motion start point / end point determination unit 3cb, the time point when looking back is the start point, and the head of the person 21 is half the head (face size) compared to the start point. A special effect is applied with the end point. The end point timing may be about １ ／ or ¼, in addition to half of the start point, as long as there is no sense of incongruity even when the image processing is finished when watching a movie. Good.

  In this embodiment, as shown in FIG. 4B, no special effect is applied to the image 33a before the start point. Separation is applied to the images 33b and 33c after the start point, and a special effect is applied such that the image 33d is brightened so as to disappear with respect to the subsequent image 33d, and a frame is added to the last image 33e. By performing such image processing, the sadness that the person 21 leaves can be expressed. In this case, the effects are not given all at once, but are changed in order. In addition to the sepia tone, other image processing such as a monochrome tone may be performed.

  The example shown in FIG. 5 is another predetermined operation of the person 21 and is a case where the person 21 is turning like a dance. At this time, when moving image shooting is performed by the camera 10, images 32a to 32f constituting the moving image as illustrated in FIG. 6A can be acquired. In this case, if the image 34a in which the face portion of the person 21 faces the front is the starting point, the face portion rotates, and the image 34f that faces the front again becomes the end point.

  Therefore, in the present embodiment, paying attention to the rotation of the face of the person 21, while the face rotates once, the color of the clothes portion is changed to change the clothes image 34 a to 34 f. Has an effect. Since the start point and the end point can be determined by analyzing the change in the movement of the face, such a change can be added. When the person 21 rotates many times, the color of the clothes may be changed similarly as one section from the front to the next front. In addition to the color, the pattern may be changed, and the pattern and the color may be changed at the same time. Further, in FIG. 6B, only the skirt is changed, but the outerwear may also be changed.

  Thus, in the present embodiment, the start point and end point of the operation are determined by paying attention to the rotation of the face of the person 21. A method for determining the change in the orientation of the face will be described with reference to FIG. FIG. 7 shows the shadow of the eyes and nose and the outline of the face of the human face. That is, the ellipse indicates the outer shape of the face, and the T shape inside the outer shape of the face indicates the shadow of the eyes and nose.

  That is, FIG. 7A shows a case where the face of the person is facing the front. In this case, the nose line 37 of the person coincides with the center line 36 of the outer shape of the face. However, as shown in FIG. 7B, when the person faces the right side, the nose line 37 is shifted to the right side from the center line 36, and the difference X becomes a positive value. On the other hand, as shown in FIG. 7C, when the person faces the left side, the nose muscle line 37 is shifted to the left side from the center line 36, and the difference X becomes negative. In the present embodiment, the face outline, the shadow of the nose and the change thereof are determined by the face change determination unit 1b.

  If it is difficult to detect the shadow of the eyes and nose, first, as shown in FIG. 8 (a), after detecting the face while facing the front, tracking the temporal change of the feature points of the shadow, It can be determined whether the face is facing to the side. In the example shown in FIG. 8, the distance to the end of the line connecting both eyes changes to X1 in FIG. 8B and X2 in FIG. 8C, so that the face is facing the right side. Can be determined. In addition to the feature points of the shadows of the eyes and nose, it is possible to determine the size φ of the head by detecting the outline around the shadows of the eyes and determine whether the person is moving away based on the size φ. it can.

  FIG. 9A shows changes in feature points when the face portion is rotated. That is, the difference X between the nose muscle line 37 and the center line 36 is 0 when the face portion is facing the front (time t0). Assuming that the face portion is gradually turned to the right side, the difference X gradually increases until time t1, and the face turns backward at time t1, so the difference X disappears. Now, assuming that the face portion is rotating, the nose muscle line moves from the left side toward the front side at time t2, and thus changes as shown by a broken line.

  FIG. 9B shows a change in the head size φ. In the example of the solid line in FIG. 9B, since the size φ decreases as time t1 and t2 elapse, it can be said that the person who is the subject is moving away. On the other hand, in the example of the broken line, it can be said that the distance from the photographer does not change because the head size φ is constant even when the times t1 and t2 have elapsed.

  9 (a) and 9 (b), the example indicated by the solid line disappears after the shadow of the face's eyes and nose is directed backward, and the size φ of the head decreases with time. Therefore, as described with reference to FIGS. 3 and 4, it is estimated that the person looks back and moves away. On the other hand, in the example indicated by the broken line (however, including the solid line part at time t0 to t1 in FIG. 9A), the shadows of the eyes and nose appear once again from the opposite side, and the size of the head Since φ does not change with time and is constant, it is estimated that the person is rotating as described with reference to FIGS. 5 and 6.

  As described above, in the present embodiment, by detecting the shadow of the eyes and nose of the face and the size φ of the head, it is possible to detect the movement of the person and to detect the start point and the end point thereof.

  Next, the operation of the camera 10 according to the present embodiment will be described using the camera control flowchart shown in FIG. The flow in the present embodiment, including the flowcharts shown in FIGS. 11, 13, and 14 described later, is executed by the control unit 1 in accordance with a program stored in a memory (not shown). In the present embodiment, the case where the moving image shooting mode and the moving image art mode are selected in the shooting mode will be described.

  When the camera 10 is powered on and activated, the flow shown in FIG. 10 starts. First, it is determined whether or not the photographing mode is set (S1). The camera 10 is provided with other modes such as a shooting mode and a playback mode, and the mode selection state of the photographer is determined based on the operation state of the operation unit 9. The mode in the default state of the camera 10 is a shooting mode.

  If the result of determination in step S1 is that the shooting mode has not been selected, processing other than shooting mode is carried out (S35). Here, processing is performed according to a mode selected by the user, such as a playback mode. When processing other than the shooting mode is performed, the process returns to step S1.

  If the result of determination in step S <b> 1 is shooting mode, imaging, display, and face detection are performed (S <b> 3). In this step, the image data acquired by the imaging unit 2 is subjected to image processing in the image processing unit 3, and live view display is performed on the display unit 8 using the image data subjected to the image processing. The photographer can determine the composition and the photo opportunity while viewing the live view display. Further, the face detection unit 2a detects whether or not a face portion is included in the image and, if included, the position and size thereof.

  Once imaging, display, and face detection have been performed, it is next determined whether or not imaging has started (S5). Here, it is determined whether or not a moving image button or the like is operated and an instruction to start moving image shooting is given. If the result of this determination is that shooting has not started, processing returns to step S1. When the shooting mode is selected, the live view display is updated while steps S1 to S5 are repeatedly executed.

  If the result of determination in step S5 is that shooting has started, then continuous shooting and temporary recording of face detection results are started (S7). Here, the image data of the moving image output from the imaging unit 2 and the face detection result output from the face detection unit 2a are temporarily recorded sequentially in the memory 0 to the memory n of the temporary recording unit 3a.

  Once the image data of the moving image and the face detection result are provisionally recorded, a trajectory determination is performed (S9). In this trajectory determination, the motion analysis unit 3ca performs a predetermined movement as described with reference to FIGS. 3 to 9 based on the temporarily recorded image data and the face detection result. It is determined whether or not, in other words, the locus of the face portion has a predetermined change. Details of the trajectory determination flow will be described later with reference to FIG.

  Once the trajectory is determined, it is next determined whether or not the trajectory is a predetermined change (S11). Here, based on the trajectory determination performed in step S9, the motion start point / end point determination unit 3cb determines whether or not the trajectory is a predetermined change.

  If the result of determination in step S11 is that the trajectory is not a predetermined change, it is next determined whether or not there is no change for a predetermined time (S21). In order to perform trajectory determination in step S9, data for a predetermined time is required. If the predetermined time is too short, sufficient determination cannot be made, and if the predetermined time is long, a large-capacity memory is required as the temporary recording unit 3a. However, in consideration of these, the length of the predetermined time may be selected as appropriate. In the present embodiment, for example, about 10 seconds.

  If the result of determination in step S21 is that there is no predetermined change in the trajectory for a predetermined time, then the previous image is compressed and recorded without image processing (S23). In this case, since the predetermined movement as described with reference to FIGS. 3 to 9 was not performed, the image data of the moving image is compressed as it is without performing image processing that has an effect on the moving image. Record. That is, the effect imparting unit 3bb records the image data in the recording unit 6 after compressing the image in the moving image compressing unit 3d without imparting any effect.

  On the other hand, if the result of determination in step S11 is that the trajectory is a predetermined change, next, start point end point determination is performed (S13). Here, in the case of a predetermined operation, the start point and end point of the operation are obtained. For example, in the case of a person who faces the opposite side as described in FIG. 3 and FIG. 4 and leaves away, the position where the face is directed to the opposite side is the starting point, and the head is half of the starting point. The end position is the end point. Further, when the face portion as described in FIGS. 5 and 6 is rotating, the position facing the front is set as the start point, and the position turning around once and facing the front is set as the end point. The flow of the start point / end point determination will be described in detail with reference to FIG.

  Once the start point / end point determination is performed, an effect image process is performed on the moving image between the start point and the end point, and image compression is performed (S15). When the start point and the end point are determined, image processing is performed so as to give a special effect to the image data of the moving image during that time. That is, since the moving image data between the start point and the end point is identified by the adoption determining unit 3ba, the effect applying unit 3b performs image processing on this image. When image processing is performed, image compression is performed by the moving image compression unit 3d. Of the operations in this step, the flow of the image processing portion will be described in detail with reference to FIG.

  Subsequently, the image before the start point is compressed and recorded (S17). Since the start point is determined in the start point / end point determination in step S13, in this step, the image data recorded in the temporary recording unit 3a before the start point is compressed in the moving image compression unit 3d and then recorded in the recording unit 6. Since the image data before the start point is not subjected to image processing, it is recorded in the recording unit 6 to reduce the load on the temporary recording unit 3a.

  If image compression recording before the start point is performed in step S17, or compression recording is performed without image processing in step S23, or if the result of determination in step S21 is a change during a predetermined time, Next, it is determined whether or not the photographing is finished (S19). Here, it is determined whether or not the moving image button or the like has been operated again to instruct the end of moving image shooting. If the result of this determination is that photography has not ended, processing returns to step S9.

  If the result of determination in step S19 is that shooting has been completed, the image after the end point determined in step S13 remains in the temporary recording unit 3a. Therefore, the processing is varied according to the continuation state of the predetermined operation after the end point. First, it is determined whether the same operation continues after the end point (S25). Here, it is determined by the motion analysis unit 3ca whether or not a predetermined operation continues after the end point and before the end of shooting.

  If the same operation is continued as a result of the determination in step S25, the effect image processing of the image after the end point is performed, and then compressed and recorded (S27). Here, the effect applying unit 3bb performs the same image processing as the image processing performed in step S15, and the image is compressed by the moving image compressing unit 3d and then recorded in the recording unit 6. If the scene is rotating as shown in FIG. 5 and FIG. 6, the effect of changing clothes persists.

  On the other hand, if the result of determination in step S25 is that the same operation has not continued, it is determined whether or not the time from the end point to the end of shooting is 5 seconds or longer (S29). If the result of this determination is that the operation has not continued for 5 seconds or more, the image from the end point to the end of shooting is not compressed and recorded as it is (S33). In this case, since the same operation has not been performed for 5 seconds or more, a completely different operation is started, and when a special effect is applied, an unnatural image is obtained. Record.

  If the result of determination in step S29 is that the time from the end point to the end of shooting is less than 5 seconds, as in step S27, effect image processing is performed on the image after the end point, and the image is compressed and recorded (S31). . Since the time from the end point is short, the effect image processing is continued. In step S27, S31, S33, when the image after the end point is recorded, the process returns to step S1.

  Next, details of the trajectory determination flow in step S9 will be described using the flowchart shown in FIG. If this flow is entered, it is first determined whether or not the face orientation is landscape (S41). Here, based on the detection result by the face detection unit 2a, the determination is made based on whether or not the shadow of the eyes and nose is a horizontal shadow.

  If the result of determination in step S41 is that the face orientation is landscape, it is next determined whether or not the eye-nose pattern has disappeared (S43). As described above, since the shadow of the eyes and nose disappears when the face of the person faces away from the front, the motion analysis unit 3ca makes a determination based on the shadow of the eyes and nose.

  If the eye / nose pattern disappears as a result of the determination in step S43, it is next determined whether or not the head can be determined (S45). Here, it is determined whether or not the head size φ as shown in FIG. 8C can be measured by the face detection unit 2a. If the size of the head can be determined as a result of this determination, it is next determined whether or not the head is reduced (S47). In this step, the determination is made based on the temporal change of the head size φ detected by the face detection unit 2a. It should be noted that the determination is made while observing the change with time from step S43 to S47.

  If the result of determination in step S47 is that the head has shrunk, pattern 1 is set (S49). In this case, as described with reference to FIGS. 3 and 4, this corresponds to a scene where a person looks back and moves away, and is set as a predetermined locus of pattern 1.

  On the other hand, if the result of determination in step S47 is that the head has not shrunk, it is next determined whether or not the eye-nose pattern can be detected again (S51). In step S43, it is detected that the shadow of the eyes and nose has disappeared. In this step, it is determined by the motion analysis unit 3ca whether the shadow of the eyes and nose has appeared again.

  If the result of determination in step S51 is that the eye-nose pattern can be detected again, it is set as pattern 2 (S53). In this case, as described in FIGS. 5 and 6, the scene is a scene in which a person is rotating, and is set as a predetermined locus of the pattern 2.

  On the other hand, if the result of determination in step S51 is that the eye-nose pattern cannot be detected again, it is next determined whether or not it is stationary for 2 seconds (S55). Here, it is determined whether there is no change for 2 seconds while the shadows of the eyes and nose are still disappearing. Note that 2 seconds is an exemplification, and may be shorter or longer than this, as long as it can be determined that a person is looking up at the sky obliquely as described later. Further, it may be the time for the subject person to take a breath after a certain operation, or the time for the person to stop without shaking.

  If the result of determination in step S55 is that it has been stationary for 2 seconds, pattern 3 is set (S57). In this case, the shadow of the eyes and nose remains disappeared, the head is not reduced in size, and remains stationary for 2 seconds. Set as a predetermined trajectory.

  If it is determined in step S41 that the face is not moving sideways, it is next determined whether or not the head is moving up and down (S61). Here, as shown in FIG. 12, the motion analysis unit 3ca determines whether or not the head is moving up and down based on the shadow of the eyes and nose. That is, as shown in FIG. 12A, when the face is facing the front, a horizontal line 38 connecting both eyes is determined. Then, as shown in FIGS. 12B and 12C, when the head moves up and down, differences Y1 and Y2 between the horizontal line 38 and the eye line 39 connecting both eyes are obtained, and the head is calculated based on Y1 and Y2. It is determined whether the part has moved up and down.

  If the result of determination in step S61 is that the head has moved up and down, it is next determined whether or not it has been stationary for 2 seconds, as in step S55 (S63). Here, it is determined whether there is no change for 2 seconds while the shadows of the eyes and nose are still disappearing. Specifically, it is checked whether the differences Y1 and Y2 between the horizontal line 38 and the eye line 39 described above are not changed. Note that 2 seconds is an example, and it may be a time that can be determined to be when facing down or when nodding. Further, it may be the time for the subject person to take a breath after a certain operation, or the time for the person to stop without shaking.

  If the result of determination in step S63 is that it has been stationary for 2 seconds, pattern 4 is set (S65). In this case, since the head moves up and down and remains as it is for 2 seconds, the scene is depressed or nodded, and the predetermined locus of the pattern 4 is set.

  In step S49, S53, S57, or S65, if any of pattern 1 to pattern 4 is set, the trajectory is set to be predetermined (S69), and the process returns to the original flow. In this case, when returning to the original flow, it is determined No in the next step S11, and the process proceeds to step S21.

  As a result of the determination in step S43, when the eye-nose pattern has not disappeared, or as a result of the determination in step S45, the head determination is impossible, or as a result of the determination in step S55, it has not been stationary for 2 seconds. If the result of the determination in step S61 is that the head has not moved up or down, or if the result of determination in step S63 is that the head has not been stationary for 2 seconds, the trajectory is set to other than the predetermined (S67). Return to the original flow. In this case, when returning to the original flow, it is determined Yes in the next step S11, and the process proceeds to step S13.

  Thus, in the trajectory determination subroutine, the motion analysis unit 3ca determines the movement of the face portion of the person, and determines whether or not the trajectory is a predetermined trajectory. In the case of a predetermined trajectory, the trajectory patterns are classified by 1 to 4.

  Next, details of the start point / end point determination flow in step S13 will be described using the flowchart shown in FIG. If this flow is entered, it is first determined whether or not it is pattern 1 (S71). In step S49 of the trajectory determination subroutine shown in FIG. 11, if a person has followed the trajectory of pattern 1, that fact is set. Therefore, in this step, it is determined whether or not pattern 1 is set.

  If the result of determination in step S71 is pattern 1, the start of face orientation change is set as the start point and the reduction start is ended (S81). As described with reference to FIGS. 3 and 4, the pattern 1 is a scene in which a person who swings to the opposite side and leaves in the direction of moving away is photographed. For this reason, the time when the orientation of the face starts to change by swinging the opposite side is set as the start point, and the time when the head is reduced in half is set as the end point.

  If the result of determination in step S71 is not pattern 1, it is next determined whether or not it is pattern 2 (S73). In step S53, if the person has followed the trajectory of pattern 2, that fact is set. Therefore, in this step, it is determined whether or not the pattern 2 is set.

  If the result of determination in step S73 is pattern 2, the front from the front is set as the start point and the end point is set (S83). As described with reference to FIGS. 5 and 6, the pattern 2 is a scene for photographing a person who is rotating to dance. For this reason, the time when it faces the front is the starting point, and the time when it turns one turn after that and turns to the front again is the end point.

  If the result of determination in step S73 is not pattern 2, it is next determined whether pattern 3 or not (S75). In step S57, if the person is following the trajectory of pattern 3, that fact is set. Therefore, in this step, it is determined whether or not the pattern 3 is set.

  If the result of determination in step S75 is pattern 3, the start of face direction change is set as the start point and change stop is set as the end point (S85). As described above, the pattern 3 is a scene for photographing a person who looks up at the sky diagonally upward. For this reason, the time when the orientation of the face starts to change is set as the start point, and the time when the change stops is set as the end point.

  If the result of determination in step S75 is not pattern 3, it is next determined whether or not the stop is in front (S77). Here, it is the case of the pattern 4, and in this case, it is a depressed state or a nodding state. Therefore, the motion analysis unit 3ca determines whether or not the face belongs to the pattern 4 and faces the front when the facial motion stops.

  If the result of determination in step S77 is that the stop is in front, the start of change in direction is taken as the start point and the stop of facial expression change is taken as the end point (S87). In this case, for example, the person who was looking down and laughed smiled at the front. Therefore, the time when the face starts to move in the front direction (that is, the change start time of the orientation) is set as the start point, and the time when the facial expression change is stopped facing the front (when the smile is smiled) is set as the end point.

  If the result of determination in step S77 is that the stop is not front, the front facial expression change stop time is taken as the start point and the time point when head movement stops is taken as the end point (S79). In this case, for example, a person who is facing the front suddenly changes his expression and looks down. Therefore, the time when the facial expression changes is set as the start point, and the time when the downward movement is stopped is set as the end point.

  In step S79, S81, S83, S85, or S87, when the start point and the end point are determined, the process returns to the original flow. As described above, in the start point / end point determination subroutine, the timing (start point and end point) for applying a special effect is determined in correspondence with a predetermined operation.

  Various processes are performed in steps S15, S27, and S31. The details of the flow of the effect image process will be described with reference to the flowchart shown in FIG. If this flow is entered, it is first determined whether or not it is pattern 1 as in step S71 (S91).

  If the result of determination in step S91 is pattern 1, image processing is performed so that the image between the start point and end point is in sepia and soft focus (S101). As described above, pattern 1 is a scene in which a person goes away from the camera 10 and moves away. Therefore, the color tone is converted into a sepia tone so as to obtain a screen suitable for this atmosphere, and the software 1 Adjust the screen contrast so that it is in focus. Note that monochrome tone may be used instead of sepia tone, and other art filters may be applied instead of soft focus.

  On the other hand, if the result of determination in step S91 is not pattern 1, it is next determined whether or not it is pattern 2 as in step S73 (S93). If the result of this determination is pattern 2, image processing is performed so as to change the color of the clothing part under the face in the image between the start point and the end point (S103). As described above, since the pattern 2 is a scene that is rotating so that the person dances as described above, the image processing is performed to change the color of the clothes of the person so that the person can enjoy and appreciate it. In addition to changing a plurality of colors, a pattern or the like may be changed and enjoyed.

  If the result of determination in step S93 is not pattern 2, it is next determined whether or not it is pattern 3 as in step S75 (S95). If the result of this determination is pattern 3, image processing is performed so that a gloss effect is produced on the image between the start point and end point (S105). As described above, Pattern 3 is a scene for shooting a person who looks up at the sky diagonally, so a glossy effect that sparkles part of the face so that the screen is suitable for this atmosphere. give. As this glittering glossing process, a process of changing the luminance of the pixels with time by displaying some of the pixels in the face portion or in the vicinity brightly or darkly is performed.

  If the result of determination in step S95 is not pattern 3, it is next determined whether or not the face is facing front in a stopped state (S97), as in step S77. If the result of this determination is that the face is facing forward in the stopped state, image processing that increases the brightness is then performed on the image between the start point and end point (S107). In this case, as described above, for example, it is a scene in which a person who was facing down and smiled smiled facing the front. Therefore, image processing is performed so as to obtain a bright screen suitable for this atmosphere.

  On the other hand, if the result of determination in step S97 is that the face is not facing forward in the stopped state, next, image processing is performed to reduce the brightness of the image between the start point and end point (S99). In this case, for example, it is a scene in which a person who is facing the front suddenly changes facial expressions and looks down. Therefore, image processing is performed to reduce the brightness of the screen so that the screen is suitable for this atmosphere and is heavy.

  When the image processing in steps S99 to S107 is performed, the process returns to the original flow. In this way, in the effect image processing subroutine, image processing is performed so that an appropriate screen is obtained in accordance with the patterned predetermined operation.

  As described above, in one embodiment of the present invention, the effect applying unit 3b (steps S15 and S27) is selected according to the movement of the subject analyzed by the motion analysis unit 3ca (track determination in step S9, see FIG. 11). , S31, see the effect image processing in FIG. 14) is performing the effect image processing on the image data. For this reason, image processing is changed according to the overall movement of the subject, and an image rich in change can be obtained. With conventional cameras, when shooting a moving subject, you can only frame it to capture its movement or adjust the timing of the start of shooting, and during the short-term movement of the subject, It was impossible for the photographer to perform various camera operations. Unlike shooting for a long time, such as when shooting a race like a competition, it was impossible to operate the camera in response to a gesture that a person performed instantaneously. In this embodiment, even with such an instantaneous operation, the camera can analyze the movement of the person and perform image processing according to the movement.

  In the embodiment of the present invention, patterns 1 to 4 are targeted as predetermined motions of the subject, but other motions may be added, and any of patterns 1 to 4 is omitted. You may do it.

  Further, in one embodiment of the present invention, in determining whether or not a predetermined operation is being performed, the determination is made based on the orientation of the face and the size of the head. Of course, the determination may be made using other information such as facial expressions.

  Furthermore, in the embodiment of the present invention, different effect image processing is set as the effect image processing according to the patterns 1 to 4, but the content of the effect image processing may be appropriately changed.

  Furthermore, in the embodiment of the present invention, the digital camera is used as the photographing device. However, the camera may be a digital single lens reflex camera or a compact digital camera, such as a video camera or a movie camera. It may be a camera for moving images, or may be a camera built in a mobile phone, a personal digital assistant (PDA), a game device, or the like. In any case, the present invention can be applied to any device capable of shooting a moving image.

  The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, you may delete some components of all the components shown by embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1 ... Control part, 1a ... Recording control part, 1b ... Face change determination part, 1c ... Mode setting part, 2 ... Imaging part, 2a ... Face detection part, ... Image processing unit, 3a ... temporary recording unit, 3b ... effect applying unit, 6 ... recording unit, 7 ... clock unit, 9 ... clock unit, 10 ... camera, 21. .. person, 31a-31e ... image, 32a-32f ... image, 33a-33e ... image, 34a-34f ... image, 36 ... center line, 37 ... nose muscle line, 38 ... horizontal line, 39 ... gaze

Claims (9)

An imaging unit that captures a subject image and outputs image data;
A face change determination unit that determines a start time and an end time of a change in the orientation of a face portion of a person image based on the image data;
An image processing unit that performs image processing of the image data based on the start time and the end time determined by the face change determination unit;
An imaging apparatus comprising: The face change determination unit further determines a change in the size of the face part,
The image processing unit determines the content of the image processing based on a change in the size of the face;
The photographing apparatus according to claim 1, wherein: An imaging unit that captures a subject image and outputs video data;
A motion analysis unit that analyzes the motion of the subject based on the image data;
An effect applying unit that performs an effect image process on the image data according to the motion of the subject analyzed by the motion analysis unit;
A recording unit for recording the image data subjected to the effect image processing by the effect applying unit;
An imaging apparatus comprising:   The apparatus according to claim 3, further comprising: an operation start point end point determination unit that determines a range on which the effect image processing is performed by the effect applying unit based on the movement of the subject analyzed by the motion analysis unit. Shooting equipment.   5. The photographing apparatus according to claim 4, wherein the effect applying unit performs the effect image processing on the image data between the start point and the end point determined by the operation start point / end point determination unit.   The photographing apparatus according to claim 3, wherein the motion analysis unit classifies the motion of the subject into a predetermined pattern.   The photographing apparatus according to claim 6, wherein the effect applying unit changes the effect image processing in accordance with the pattern by the motion analysis unit.   The photographing apparatus according to claim 3, wherein the motion analysis unit analyzes the movement of the subject using at least one of the direction and size of the face of the person of the subject. Capture the subject image and output the video image data.
Based on the above image data, analyze the movement of the subject,
In accordance with the analyzed movement of the subject, effect image processing is performed on the image data,
Record the image data subjected to the effect image processing;
An image data generation method characterized by the above.