JP2014112787A - Video processing device and video processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To create a digest of video reflecting the intension of an operator accurately.SOLUTION: A video processing device includes a CMOS element 160 for capturing the image of an operator, a face analysis unit 220 for analyzing the features of face of the operator from a first video captured by the CMOS element 160, an expression evaluation value calculation unit 222 for calculating an evaluation value by quantifying the expression of face of the operator based on the analysis results from the analysis unit 220, from the first video captured while capturing or reproducing a second video different from the first video, a record file creation unit 228 for recording the evaluation value thus quantified on the same time line as the second video, and a video edition unit 226 for creating a digest by extracting the partial videos of the second video sequentially based on the evaluation value.

Description

  The present invention relates to a moving image processing apparatus and a moving image processing method.

  Conventionally, a technique for creating a digest version of a moving image is known. For example, in Patent Document 1 below, human relation information is stored, and in addition to the person specified by the user, an image showing a person closely related to the person is extracted to create a digest image. An information processing apparatus is described.

  Japanese Patent Application Laid-Open No. 2004-228688 describes a technique for detecting a change in a specific image and sound as an event and generating a digest version video with a variable playback speed. Patent Document 3 below describes a system for setting a favorite degree of an image being browsed based on whether or not the viewer is smiling when playing back a still image with a digital camera or the like. .

JP 2011-82915 A JP 2010-39877 A JP 2012-156750 A

  In recent years, electronic devices such as a digital still camera and a smart phone are also equipped with a moving image recording function. Video playback is suitable for expressing a sense of realism, but has a drawback that it takes time to confirm the content by playback compared to a still image. In moving image reproduction, there are many redundant portions other than the main scene. For this reason, even if a moving image is recorded, the reproduction frequency of the moving image tends to be low.

  For this reason, a function for automatically automatically creating (automatically editing) a digest (summary) moving image is required. However, in the techniques described in Patent Literature 1 and Patent Literature 2, since the digest is created by detecting the characteristics of the person and the sound to be emitted, the application status of the digest creation is roughly limited by the characteristics and voice of the person. Therefore, it has been difficult to create a desired digest. In addition, in the techniques described in Patent Documents 1 and 2 and the like, there is no guarantee that the specified person appears in the scene to be left in the digest, and there is no guarantee that the voice is inserted in the scene to be left. It is difficult to create. Furthermore, in the case where a designated person always appears in a moving image, there is a problem that an important scene cannot be determined and it is difficult to generate a digest.

  In addition, the method described in Patent Document 3 recognizes whether or not the viewer is smiling, but is a technique for setting a favorite degree of a still image, and is assumed to create a digest of a moving image. It was not a thing.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a new and improved video digest that accurately reflects the intention of the operator. Another object is to provide a moving image processing apparatus and a moving image processing method.

  In order to solve the above problems, according to an aspect of the present invention, a first moving image capturing unit that captures an image of an operator and a face of the operator from the first moving image captured by the first moving image capturing unit. An analysis unit for analyzing the characteristics of the operator, and the face of the operator based on the analysis result by the analysis unit from the first moving image captured at the time of capturing or reproducing the second moving image different from the first moving image An evaluation value calculation unit that calculates an evaluation value by digitizing the facial expression, a recording unit that records the digitized evaluation value on the same timeline as the second moving image, and the second value based on the evaluation value There is provided a moving image processing apparatus including a moving image editing unit that sequentially extracts partial moving images of moving images and generates a digest.

  According to the above configuration, the features of the operator's face are analyzed from the first video, and the facial features are captured from the first video captured at the time of capturing or reproducing the second video different from the first video. Based on the analysis result, the facial expression of the operator is digitized and an evaluation value is calculated. The numerical evaluation value is recorded on the same timeline as the second moving image, and a partial moving image of the second moving image is sequentially extracted based on the evaluation value to create a digest. Therefore, the facial expression evaluation value for creating the digest is calculated at the time of shooting or playback of the second video, so that the emotion of the operator can be accurately reflected in the evaluation value and a desired digest can be created. It becomes possible to do.

  In addition, a second moving image capturing unit that captures the second moving image is provided, and the evaluation value calculating unit performs the numerical value when the second moving image capturing unit captures the second moving image. According to this configuration, the expression evaluation value can be digitized based on the second moving image captured by the second moving image capturing unit.

  The moving image editing unit sequentially extracts the partial moving images in a section where the value of the evaluation value is larger than a predetermined threshold value. According to this configuration, since the partial moving images are sequentially extracted in the section where the evaluation value is larger than the predetermined threshold value, the partial moving images are sequentially extracted when the change in the facial expression of the operator is large. This makes it possible to create a desired digest.

  In addition, the moving image editing unit extracts the partial moving image by changing the threshold value so that the total time of the partial moving image falls within a predetermined time. According to this configuration, since the threshold value is changed so that the total time of the partial moving image falls within a predetermined time, the partial moving image can be extracted with a desired length.

  Further, the moving image editing unit sequentially extracts the partial moving images in a section obtained by adding a predetermined margin time before or after the section in which the value of the evaluation value is larger than a predetermined threshold value. According to this configuration, a desired digest can be created even when the sampling interval of evaluation values is relatively large.

  Further, the evaluation value calculation unit performs the digitization based on a predetermined rule. According to this configuration, since the digitization is performed based on a predetermined rule, the evaluation value can be calculated based on the predetermined rule.

  Further, the rule is defined so that the weight of the evaluation value is increased when the facial expression changes drastically. According to this configuration, when the change in facial expression is severe, the evaluation value is highly weighted, so that the change in expression can be accurately reflected in the evaluation value.

  The recording unit records the evaluation value as metadata in the data of the second moving image. According to this configuration, since the evaluation value is recorded as metadata in the data of the second moving image, the evaluation value and the data of the second moving image can be recorded in association with each other.

  In addition, the recording unit records the evaluation value as separate data that has a one-to-one correspondence with the data of the second moving image. According to this configuration, since the evaluation value is recorded as separate data that has a one-to-one correspondence with the second moving image data, the evaluation value and the second moving image data can be recorded in association with each other.

  In order to solve the above-described problem, according to another aspect of the present invention, the step of imaging the operator and the characteristics of the operator's face from the first moving image captured by the first moving image capturing unit are obtained. Analyzing the facial expression of the operator based on the analysis result of the facial features from the first moving image captured at the time of imaging or playback of the second moving image different from the first moving image Numerically calculating the evaluation value, recording the numerical evaluation value on the same timeline as the second moving image, and sequentially sub-moving the second moving image based on the evaluation value A moving image processing method comprising: extracting and generating a digest.

  According to the above configuration, the features of the operator's face are analyzed from the first video, and the facial features are captured from the first video captured at the time of capturing or reproducing the second video different from the first video. Based on the analysis result, the facial expression of the operator is digitized and an evaluation value is calculated. The numerical evaluation value is recorded on the same timeline as the second moving image, and a partial moving image of the second moving image is sequentially extracted based on the evaluation value to create a digest. Therefore, the facial expression evaluation value for creating the digest is calculated at the time of shooting or playback of the second video, so that the emotion of the operator can be accurately reflected in the evaluation value and a desired digest can be created. It becomes possible to do.

  In addition, the method includes a step of imaging the second moving image, and in the step of calculating the evaluation value, the numerical value is performed when the second moving image is captured. According to this configuration, the evaluation value of the facial expression can be quantified based on the second moving image captured in the step of capturing the second moving image.

  In the step of generating the digest, the partial moving images in a section where the value of the evaluation value is larger than a predetermined threshold value are sequentially extracted. According to this configuration, since the partial moving images are sequentially extracted in the section where the evaluation value is larger than the predetermined threshold value, the partial moving images are sequentially extracted when the change in the facial expression of the operator is large. This makes it possible to create a desired digest.

  Further, in the step of generating the digest, the partial moving image is extracted by changing the threshold value so that the total time of the partial moving image falls within a predetermined time. According to this configuration, since the threshold value is changed so that the total time of the partial moving image falls within a predetermined time, the partial moving image can be extracted with a desired length.

  In the step of generating the digest, the partial moving images in a section obtained by adding a predetermined margin time before or after the section in which the value of the evaluation value is larger than a predetermined threshold are sequentially extracted. According to this configuration, a desired digest can be created even when the sampling interval of evaluation values is relatively large.

  In the step of calculating the evaluation value, the quantification is performed based on a predetermined rule. According to this configuration, since the digitization is performed based on a predetermined rule, the evaluation value can be calculated based on the predetermined rule.

  Further, the rule is defined so that the weight of the evaluation value is increased when the facial expression changes drastically. According to this configuration, when the change in facial expression is severe, the evaluation value is highly weighted, so that the change in expression can be accurately reflected in the evaluation value.

  In the recording step, the evaluation value is recorded as metadata in the data of the second moving image. According to this configuration, since the evaluation value is recorded as metadata in the data of the second moving image, the evaluation value and the data of the second moving image can be recorded in association with each other.

  In the recording step, the evaluation value is recorded as separate data corresponding to the data of the second moving image on a one-to-one basis. According to this configuration, since the evaluation value is recorded as separate data that has a one-to-one correspondence with the second moving image data, the evaluation value and the second moving image data can be recorded in association with each other.

  According to the present invention, it is possible to create a video digest that accurately reflects the operator's intention.

It is a schematic diagram which shows a mode that a to-be-photographed object is image | photographed with electronic devices, such as a smart phone and a digital camera. It is a schematic diagram which shows the image which the electronic device image | photographed. 1 is a schematic diagram illustrating a configuration of an imaging apparatus 100 according to an embodiment of the present invention. It is a schematic diagram for demonstrating the rule which prescribes | regulates an evaluation value. It is a schematic diagram for demonstrating the rule which prescribes | regulates an evaluation value. It is a schematic diagram which shows the automatic edit function by a moving image edit part. When the evaluation values of the elements (a) to (e) shown in FIG. 5 and the total value (sum) of the evaluation values change with the passage of time (time), a section for extracting the digest video is shown. It is a schematic diagram shown. It is a schematic diagram which shows the example which plotted the relationship between the sampling time (Sampling Time) of an evaluation value, and an evaluation value, and linearly interpolated between sampling time. It is a schematic diagram which shows the preservation | save method of an evaluation value. It is a flowchart which shows the process of the moving image processing method in an imaging device.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  In a portable electronic device such as a recent smartphone (SmartPhone), a camera for taking a self-portrait (in-camera) is mounted on the operator side separately from a camera for recording a subject. Has appeared. A configuration including a camera for self-portrait is expected to be extended to various devices including a portable device such as a digital still camera (DSC).

  Imaging devices such as digital still cameras are mainly downsized due to the progress of miniaturization, so that a photographer visually recognizes a subject on a liquid crystal display screen (LCD) without a viewfinder like a smartphone. . In this case, since the photographer shoots the face away from the imaging device by a certain distance, if the in-camera is provided, the entire face of the photographer (operator) can be photographed.

  FIG. 1 is a schematic diagram illustrating a state in which a subject is photographed by an electronic device 300 such as a smartphone or a digital camera. The electronic device 300 captures a subject with the front camera 102 based on the operation of the photographer. The electronic device 300 includes an in-camera 304 on the side opposite to the camera 302 (photographer side), and the user's face can be captured by the in-camera 304.

  FIG. 2 is a schematic diagram illustrating an image captured by the electronic device 300. Here, FIG. 2B shows a state in which an image taken by the camera 302 in front of the electronic device 300 is marked on the LCD 306 of the electronic device 300. FIG. 2A shows an image taken by the in-camera 304 of the electronic device 300. As shown in FIG. 2A, the photographer is photographed by the in-camera 304. As described above, the electronic device 100 including the in-camera 304 in addition to the normal camera 302 can photograph the photographer's face while photographing the subject.

  Images captured by the camera 302 are various subjects such as a person, a landscape, a vehicle, and a building, and the camera 302 does not always detect a person's face. On the other hand, in most cases, the image captured by the in-camera 304 is an image of the face of the photographer who is looking directly at the LCD 306 disposed in the vicinity of the in-camera 304. Therefore, according to the in-camera 304, it is possible to capture the almost front face of the photographer who is looking straight at the LCD 306. Further, since the light from the LCD 306 is used as an illumination even at low illumination, the photographer's face can be captured by the in-camera 304 even in the dark. Therefore, according to the electronic device 300 provided with the in-camera 304, the photographer's face can be always captured, and face detection and facial expression detection can be performed.

  In the present embodiment, the in-camera 304 that can always capture the photographer's (operator's) face is used to automatically edit the moving image based on the photographer's face information (particularly facial expression information). . Details will be described below.

  FIG. 3 is a schematic diagram showing a configuration of the imaging apparatus 100 as an embodiment as a moving image processing apparatus according to the present invention. FIG. 3 is a block diagram mainly related to an image processing pipeline (Pipe Line) in the digital still camera according to the present embodiment. As shown in FIG. 3, an imaging apparatus 100 according to an embodiment of the present invention includes a zoom lens (group) 102, a diaphragm 104, a focus lens (group) 108, a lens CPU 110, drivers 112 and 114, Motors 116 and 118, A / D conversion circuit 120, ROM 122, RAM and 124 are provided.

  In addition, the imaging apparatus 100 includes a shutter 126, a driver 128, a motor 130, a complementary metal oxide semiconductor (CMOS) element 132 as an imaging element, a CDS (Correlated Double Sampling) circuit 134 integrated with an amplifier, an A / A D converter 136, an image input controller 138, and a CPU 200 are provided.

  The imaging apparatus 100 also includes a nonvolatile memory 140, a compression processing circuit 142, an LCD driver 144, an LCD (Liquid Crystal Display) 146, a media controller 150, a recording medium 152, and a VRAM (Video Random Access Memory). 154, a memory (SDRAM) 156, an operation member 158, and a battery 148.

  The diaphragm 104 and the focus lens 108 are driven via motors 116 and 118 controlled by the drivers 112 and 114, respectively. The zoom lens 102 is a lens that is moved back and forth in the optical axis direction and continuously changes the focal length. The diaphragm 104 adjusts the amount of light incident on the CMOS element 132 when capturing an image. The shutter 126 controls the exposure time for the CMOS element 132 when an image is captured. The focus lens 108 is moved back and forth in the optical axis direction, and adjusts the focus of the subject image formed on the CMOS element 132.

  The CMOS element 132 is an element for converting light incident through the zoom lens 102, the stop 104, and the focus lens 108 into an electric signal.

  The CDS circuit 134 is a circuit in which a CDS circuit that is a kind of sampling circuit that removes noise from the electrical signal output from the CMOS element 132 and an amplifier that amplifies the electrical signal after removing the noise are integrated. . In the present embodiment, a circuit in which the CDS circuit and the amplifier are integrated is used. However, the CDS circuit and the amplifier may be configured as separate circuits.

  The A / D converter 136 converts the electrical signal generated by the CMOS element 132 into a digital signal, and generates raw image data (raw data, image data). The image input controller 138 controls input of raw image data (image data) generated by the A / D converter 136 to the recording medium 152.

  In addition, the imaging apparatus 100 according to the present embodiment includes a second CMOS element 160 that functions as an in-camera, an amplifier-integrated CDS circuit 162, an A / D converter 164, and an image input controller 166. . The CDS circuit 162, the A / D converter 164, and the image input controller 166 are provided for the CMOS element 160. In addition, the imaging apparatus 100 includes an optical lens 168 provided for the CMOS element 160. The optical axis of the optical lens 162 is directed to an operator who operates the imaging apparatus 100. The optical lens 162 uses the operator's face as a subject, and the subject image (image of the operator's face) is a second CMOS element. The image is formed on 160 imaging surfaces. The CMOS element 160 photoelectrically converts the subject image and inputs it to the image input controller 166. An in-camera is configured by these components.

  The CDS circuit 162 is a circuit in which a CDS circuit, which is a kind of sampling circuit, that removes noise from the electrical signal output from the CMOS element 160 and an amplifier that amplifies the electrical signal after removing the noise are integrated. .

  The A / D converter 164 converts the electrical signal generated by the CMOS element 160 into a digital signal, and generates raw image data (raw data, image data). The image input controller 166 controls input of raw image data (image data) generated by the A / D converter 164 to the recording medium 152.

  The non-volatile memory 140 is a memory that stores data that is always held by the imaging apparatus 100. The nonvolatile memory 140 can store a program for causing the CPU 200 to function.

  The compression processing circuit 142 performs compression processing for compressing image data output from the CMOS elements 132 and 136 into image data of an appropriate format. The image compression format may be a reversible format or an irreversible format. As an example of an appropriate format, it may be converted into a JPEG (Joint Photographic Experts Group) format or JPEG2000 format.

  The LCD 146 performs live view display before performing an imaging operation, various setting screens of the imaging apparatus 100, display of captured images, playback of moving images, and the like. Display of image data and various types of information of the imaging apparatus 100 on the LCD 146 is performed via the LCD driver 148.

  The memory (SDRAM) 156 temporarily stores an image captured by the CMOS element 132 and the CMOS element 160. The recording medium 152 has a storage capacity sufficient to store a plurality of images. Reading and writing of images to and from the memory (SDRAM) 156 is controlled by the image input controller 138 and the image input controller 166.

  The VRAM 154 holds the content displayed on the LCD 146, and the resolution and the maximum number of colors of the LCD 146 depend on the capacity of the VRAM 154.

  The recording medium 152 records an image captured by the CMOS element 132 and the CMOS element 160. Input / output to / from the recording medium 152 is controlled by the media controller 150. As the recording medium 152, a memory card, which is a card-type storage device that records data in a flash memory, can be used.

  The CPU 200 issues a signal system command to the CMOS elements 132 and 160, the CDS circuits 134 and 162, and performs an operation system command corresponding to the operation of the operation member 158. In the present embodiment, only one CPU is included, but a signal-related command and an operation-related command may be executed by separate CPUs.

  The CPU 200 includes an AE / AWB / AF evaluation value calculation unit 202, an appropriate AWB calculation unit 204, an image processing unit 206, an AF calculation / control unit 208, an AE calculation / control unit 210, a GUI management unit 212, A timing generator (TG1) 214, an I / O 216, an SIO 218, and a timing generator (TG2) 219 are provided. The CPU 200 also includes a face recognition unit 220, a facial expression evaluation value calculation unit 222, a reproduction processing unit 224, a moving image editing unit 226, and a recording file generation unit 228.

  The AE / AWB / AF evaluation value calculation unit 202 uses the image data output from the CMOS elements 132 and 160 as an AE evaluation value as exposure amount information, an AWB evaluation value as white balance information, and an AF evaluation as AF information. Value. The appropriate AWB calculation unit 204 performs processing for calculating an appropriate white balance value. The image processing unit 206 performs processing such as light amount gain correction, image edge processing (outline enhancement processing), and white balance adjustment on the image data output from the CMOS element 132.

  The AF calculation / control unit 208 determines the driving amount of the focus lens 108 when shooting the subject based on the AF evaluation value. The lens CPU 110 controls the driver 114 based on the determined driving amount and drives the motor 118. As a result, the focus lens 108 is driven to the in-focus position.

  In addition, the AE calculation / control unit 210 determines an aperture value and a shutter speed when photographing a subject based on the AE evaluation value. CPU 200 controls driver 128 based on the determined shutter speed to drive motor 130. Thereby, the shutter 126 is driven. Further, the lens CPU 110 controls the driver 112 based on the determined aperture value and drives the motor 116. Thereby, the diaphragm 104 provided in the lens is driven.

  When the operation member 158 is operated by the user, the GUI management unit 212 receives operation input information from the operation member 158. The CPU 140 performs various processes based on operation input information from the operation member 158 received by the GUI management unit 212. For example, when the GUI management unit 158 receives operation input information for generating a digest video from the operation member 158, the video editing unit 226 performs a process for generating a digest video.

  The timing generator (TG1) 214 inputs a timing signal to the CMOS element 132. That is, the driving of the CMOS element 132 is controlled by the timing signal from the timing generator (TG1) 214. The timing generator (TG1) 214 can cause the CMOS element 132 to have an electronic shutter function by causing the image light from the subject to enter during the time that the CMOS element 132 is driven.

  Similarly, the timing generator (TG2) 219 inputs a timing signal to the CMOS element 160. That is, the driving of the CMOS element 160 is controlled by the timing signal from the timing generator (TG2) 219. The timing generator (TG2) 219 can cause the CMOS element 160 to have an electronic shutter function by causing the image light from the subject to enter during the time that the CMOS element 160 is driven.

  The RGB image signal obtained from the CMOS element 132 shown in FIG. 3 is subjected to image front process processing such as defective pixel correction and black level correction mainly in the image processing unit 206, and further white balance is performed. Image recording is performed by performing electronic processing such as (White Balance) correction processing, Bayer color interpolation (Demosaic) processing, color correction processing, and gamma (Gamma) correction processing. Each functional block shown in FIG. 3 can be configured by a circuit (hardware) or a central processing unit (CPU) and a program (software) for causing it to function. Can be stored in a recording medium such as a non-volatile memory 140 or a memory connected from the outside.

  As described above, an object (person, landscape, etc.) taken by a photographer as a subject is converted into image data by the CMOS element 132. On the other hand, the photographer's face is converted into image data by the CMOS element 160.

  In the present embodiment, the CMOS elements 132 and 160 are used as the imaging elements. However, the present invention is not limited to this example, and other image sensors such as CCD elements are used instead of the CMOS elements 132 and 160. Also good. Since the CMOS elements 132 and 160 can convert the video light of the subject into an electrical signal at a higher speed than the CCD element, it is possible to shorten the time from when the subject is imaged to when the image is combined.

  As shown in FIG. 1, the imaging apparatus 100 includes a body main body 250 and an interchangeable lens 260, and the interchangeable lens 260 can be removed from the body main body 250. On the other hand, in the imaging apparatus 100, the body main body 250 and the interchangeable lens 260 may be configured integrally.

  In the imaging apparatus 100 configured as described above, by monitoring the photographer's facial expression with the in-camera, it is possible to infer the photographer's feelings about the image being captured or the image being reproduced. It is. For example, in a situation where parents take pictures of a child's presentation or athletic meet, recording is often started with a sufficient allowance before the child appears so as not to miss the shot. When a child who wants to take a picture appears or a performance begins, the expression changes naturally and sometimes the voice is spoken. Then, by recognizing changes and states of facial expressions, the photographer's “subjective feeling” can be determined “objectively” for each part of the moving image from the in-camera image.

  In the present embodiment, an objective numerical value is calculated according to the level of change of the emotion and emotion of the photographer (operator) determined from the facial expression captured by the in-camera. The higher the value, the more important scenes of the movie are included. Then, by sequentially extracting moving image sections having high evaluation values so as to fit in a desired time, a digest moving image composed of important portions is automatically generated.

  The evaluation value can be calculated at the time of moving image capturing by the CMOS element 132. Also, the evaluation value can be calculated when the moving image of the moving image file recorded on the recording medium 152 by the reproduction processing unit 224 is reproduced. After calculating the evaluation value, a digest video can be created at an arbitrary timing based on the evaluation value.

For this reason, the face recognition unit (analysis unit) 220 of the CPU 200 recognizes the facial expression of the photographer (or the operator who views the playback moving image) from the image data captured by the CMOS element 160 and analyzes the feature. That is, the face recognition unit 220 functions as an analysis unit that analyzes the characteristics of the operator's face.
The facial expression evaluation value calculation unit 222 calculates the evaluation value by digitizing the facial expression based on the facial expression recognized by the face recognition unit 220. The digitized evaluation value data is generated at the start of recording of the captured moving image by the CMOS element 132 and is recorded on the same timeline as the captured moving image. When calculating an evaluation value at the time of reproducing a moving image, the digitized evaluation value data is generated when the reproduction processing unit 224 starts reproducing the moving image, and is recorded on the same timeline as the reproduced moving image.

  The evaluation value of the facial expression can be created based on a preset rule. Although there are individual differences in the evaluation value of facial expressions, a relative judgment can be made by extracting from the entire image data of facial expressions for one moving image in order from the top places where expressions of emotions are expressed. Thereby, not only the individual difference of the evaluation value but also the difference of the absolute amount of the evaluation value due to the difference of the situation is reduced.

  Hereinafter, a method for calculating the evaluation value will be specifically described. FIG. 4 and FIG. 5 are schematic diagrams for explaining the rules defining the evaluation value. Here, FIG. 4 is a schematic diagram showing elements that determine an evaluation value in a facial expression of a person. As shown in FIG. 4, the evaluation values are elements such as (a) narrowing the eyes, (b) opening the eyes, (c) lowering the eyes, (d) opening the mouth, (e) increasing the mouth angle. Determined from. For example, the movement of the photographer's heart can be inferred by observing the mouth angle of (e). Therefore, based on the evaluation value, it is possible to detect the timing at which the photographer speaks even in a noisy environment.

  FIG. 5 is a schematic diagram showing rules for determining an evaluation value based on the elements (a) to (e) shown in FIG. As shown in FIG. 4, when each element (a) to (e) is at the normal level, the evaluation value is 0. On the other hand, the maximum evaluation value when each element changes is set, and the evaluation value is increased according to the level of change. For example, with respect to “(a) eyes become narrower”, the evaluation value when the eyes become the narrowest is set to 8, and the evaluation value is increased according to the level at which the eyes become thinner. Similarly, for “(b) widening eyes”, the evaluation value when the eyes are most widened is 12, and the evaluation value is increased according to the level at which the eyes are widened.

  Here, the rule shown in FIG. 5 increases the weighting for an element in which a change in the facial expression of a subject appears severely. For example, when your mouth is wide open (for example, when you are unintentionally speaking out and cheering on your child), or when your mouth is wide (for example, when you are laughing), This is largely reflected in the elements of (d) Mouth opening ”and“ (e) Mouth angle rising ”. Therefore, it is desirable to weight each element in FIG. In FIG. 5, comparing “(c) lowering of eyes” and “(d) opening of mouth”, the evaluation value when “(c) lowering of eyes” is “4” “(D) Open mouth” is “20”. This is because, when each of the two changes to the maximum, “(d) mouth open” represents a greater emotional change. Thus, by changing the weighting of the evaluation value according to each element (a) to (e), it is possible to more appropriately evaluate the user's emotion based on the evaluation value.

  For each element (a) to (e), it can be determined by setting the normal level and the maximum level in advance based on the user's facial expression to determine the normal to maximum level in FIG. it can. For example, when “(a) opening eyes” is given as an example, when the user's face is photographed for the first time (such as at the start of photographing), the distance between both eyes, the size of the eyes (width), the face outline, etc. Based on this, based on this, the normal eye size (solid line N shown in FIG. 4) and the eye size when the eye is fully opened (one point shown in FIG. 4) A chain line M) is set in advance. Then, by detecting how close the eye size is to the range of the alternate long and short dash line M with respect to the normal eye size (solid line N), the level of the eye from normal to maximum shown in FIG. It can be determined whether it is wide open.

  The face recognition unit 220 recognizes and analyzes features such as eyes, mouth, and nose from the recognized face image. The facial expression evaluation value calculation unit 222 calculates the evaluation value of each element (a) to (e) based on the feature detected by the face detection unit 220, and sums the evaluation value of each element to obtain Σ {(a) The final evaluation value is calculated by calculating + (b) + (c) + (d) + (e)}. The rules shown in FIG. 5 are stored in the non-volatile memory 140 of the imaging device 100. Further, the user can optimally change the rule shown in FIG. 5 by performing an operation from the operation member 158.

  When generating a digest movie, a digest movie can be generated by extracting a segment of a movie whose evaluation value exceeds a predetermined threshold.

  According to the above configuration, when a user visually recognizes a captured video or a playback video, if the user shows interest in the video, is impressed, or is moved, the evaluation value of the facial expression is Get higher. Therefore, it is possible to optimally generate a video digest by generating a video digest based on the evaluation value.

  In addition, when the photographer (or an operator who views the playback video) knows the rules as shown in FIG. 5 in advance, the photographer or the like can change the expression by moving the expression intentionally. It is possible to select and specify a partial moving image to be intentionally extracted. For example, if you want to include the video currently being shot (or played back) in the digest video, the evaluation value will be calculated high by intentionally making the expression smile, Can be included in the digest video. Therefore, even when quietness is required, or even when shooting silently so as not to interfere with the subject's voice or sound, without using special tools and devices, and without performing special operations, By simply changing the facial expression, it is possible to embed instruction information for automatically editing a digest video in the timeline of the video during playback or recording. In this way, the photographer knows in advance the rules for moving image extraction, so that it is possible to embed in the moving image while intentionally shooting a portion desired to be left by automatic editing. Thereby, a desired partial moving image can be extracted, and a digest moving image can be created efficiently.

  Further, since the evaluation value can be calculated simultaneously with the shooting of the moving image, it is not necessary to perform feature detection after shooting the moving image, and it is not necessary to rescan the moving image data in order to calculate the evaluation value. Therefore, the evaluation value can be calculated efficiently and quickly.

  Next, specific processing performed by the imaging apparatus 100 according to the present embodiment will be described. When the photographer photographs the subject, a moving image of the subject is captured by the CMOS element 132. At the same time, the face of the photographer (or the operator who views the playback video) is imaged by the CMOS element 160.

  The face recognition unit 220 of the CPU 200 detects the photographer's face from the moving image of the subject imaged by the CMOS element 132 and analyzes the characteristics thereof. Then, the facial expression evaluation value calculation unit 222 of the CPU 200 extracts the elements (a) to (e) shown in FIG. 5 from the photographer's face recognized by the face recognition unit 220, and the facial expression based on the rules of FIG. The evaluation value of is calculated. As described above, the facial expression evaluation value calculation unit 222 adds the evaluation values of the elements (a) to (e) and finally calculates the evaluation value. The recording file generation unit 228 records the calculated evaluation value on the same timeline as the captured moving image.

  When calculating an evaluation value during moving image reproduction, the reproduction processing unit 224 of the CPU 200 reads out and reproduces the moving image file recorded on the recording medium 152. The reproduced moving image is displayed on the LCD 146. During the moving image reproduction, the operator visually recognizes the reproduced moving image on the LCD 146. At this time, the operator's face is imaged by the CMOS element 160. Then, the face recognition unit 220 detects the photographer's face from the moving image captured by the CMOS element 160 and analyzes the feature. The facial expression evaluation value calculation unit 222 extracts the elements (a) to (e) shown in FIG. 5 from the photographer's face recognized by the face recognition unit 220, and evaluates facial expressions based on the rules of FIG. Calculate the value. The recording file generation unit 228 records the calculated evaluation value on the same timeline as the reproduced moving image. As a result, even during moving image reproduction, evaluation value data can be recorded in the timeline of the moving image, as in the case of shooting.

  The moving image editing unit 226 of the CPU 200 edits the moving image based on the facial expression evaluation value. For example, the imaging apparatus 100 operates the automatic editing function by the moving image editing unit 226 when the user wants to quickly check the content of the moving image or when the user wants to extract only the main part of the moving image. In this case, the moving image editing unit 226 performs moving image editing when an instruction for moving image editing by the operator is input to the operation member 158.

  In addition, the editing of the moving image by the moving image editing unit 226 can be automatically performed immediately after the end of shooting or immediately after the end of moving image reproduction. The editing of moving images by the moving image editing unit 226 can also be automatically performed when a list of image files is displayed on the LCD 146 as thumbnails. The digest movie generated by the movie editing is recorded on the recording medium 152.

  FIG. 6 is a schematic diagram for explaining an automatic editing function by the moving image editing unit 226. In the automatic editing by the moving image editing unit 226, moving images in a section having a large evaluation value are sequentially extracted with reference to the evaluation value of the facial expression. At this time, a section having a large evaluation value is determined based on a predetermined threshold value. In FIG. 6, as threshold values for extracting moving images, a threshold value T1 when the editing time is short, a threshold value T2 when the editing time is medium, and a threshold value T3 when the editing time is long. Three types are set.

  When the editing time is short, the moving images in the sections R11 and R12 whose evaluation values exceed the threshold value T1 are extracted to generate a digest moving image. When the editing time is intermediate, the sections R21, R22, R23, R24, and R25 where the evaluation value exceeds the threshold value T2 are extracted to generate a digest video. If the editing time is long, sections R31, R32, R33, and R34 whose evaluation values exceed the threshold value T3 are extracted to generate a digest movie (partial movie).

  Thus, when the automatic editing function is activated, the moving image editing unit 226 changes the threshold value of the evaluation value so that the editing time desired by the photographer (operator) is reached. That is, the moving image editing unit 226 searches for the optimum threshold value by changing the threshold value so that the total time of the extracted digest moving image falls within a predetermined time. Then, an evaluation value section exceeding the threshold value is extracted, a threshold value whose section total time is closest to the desired editing time is searched, and a digest movie is created by connecting the extracted sections. Thereby, the length of the digest video can be set to a length desired by the user. In FIG. 6, three threshold values T1, T2, and T3 are shown, but the threshold value can be set to any value other than T1, T2, and T3. The user can freely set the length of the digest video by operating the operation member 158. The moving image editing unit 226 optimally adjusts the threshold value based on the digest moving image length information input from the operation member 158. Thereby, the length of a digest animation can be made into a user's desired time.

  FIG. 7 shows a digest video when each evaluation value of each element (a) to (e) shown in FIG. 5 and the total value (sum) of each evaluation value change with time (time). It is a schematic diagram which shows the area to extract. When the threshold value of the evaluation value is “15”, the total (sum) of the evaluation values is 15 or more in the section from 12 seconds to 16 seconds, and therefore, from 12 seconds to 16 seconds shown in the thick frame in FIG. Is extracted as a digest video.

  FIG. 8 shows an example in which the relationship between the sampling time of the evaluation value (Sampling Time) and the evaluation value is plotted and linear interpolation is performed between the sampling times. Here, a case where the sampling time is 4 seconds is shown. The threshold value of the evaluation value is “15”. In FIG. 8, a section where the threshold value exceeds 15 is a section R4 of 8.5 seconds to 18.0 seconds. A moving image in this time zone is extracted to generate a digest moving image.

  It should be noted that if a moving image in a section where the threshold value exceeds “15” is extracted due to the sampling time, a desired moving image may deviate from the digest moving image at the beginning or the last part of the moving image. For this reason, the extraction of the moving image may be started about several seconds before the time when the threshold value exceeds “15”. A section R5 shown in FIG. 8 shows an example in which the moving image extraction start position is set at a timing approximately 2 seconds earlier than the time when the threshold value exceeds 15. Similarly, it is desirable that the end of moving image extraction is also later than the time when the threshold value is “15” or less. Thereby, the user's desired part does not leak from the digest video, and a digest video that is easy to watch can be created. The start and end images of the digest movie may be displayed with fade-in and fade-out.

  FIG. 9 is a schematic diagram illustrating a method for storing evaluation values. FIG. 9A shows a case where the evaluation value is stored as metadata in the moving image file 400. In this case, the moving image file 400 includes a header 402, a facial expression evaluation value 404, and moving image data 406.

  FIG. 9B shows a case where the evaluation value is stored as another file 500 associated with the moving image file 400. In this case, the moving image file 400 and the evaluation value file 500 are in a one-to-one correspondence by changing the extension.

  In both cases of FIG. 9A and FIG. 9B, the moving image file 400 and the evaluation value file 500 are recorded on the recording medium 152. By storing the video file 400 and the evaluation value in association with each other, the video editing unit 226 can generate a digest video at an arbitrary timing. Further, as shown in FIG. 7, since the evaluation value can be expressed by text data, the evaluation value can be surely saved only by saving the text data.

  When storing the evaluation value, it is possible to store the evaluation value in a short time by reducing the amount of data by a method such as thinning sampling appropriately or converting the evaluation value into text data. Even when editing a moving image, it is possible to read the evaluation value with a short reading operation.

  The digest movie generated by the movie editing unit 226 is also recorded on the recording medium 152. After the digest movie is generated, the digest movie can be reproduced by selecting a desired digest movie.

  For example, a thumbnail of a digest video may be displayed on the LCD 146 together with a thumbnail of a still image, and the digest video may be enlarged and reproduced to the display screen size of the LCD 146 by clicking the thumbnail of the digest video. In addition, when displaying the digest moving image as a thumbnail on the LCD 146, a method of repeatedly displaying the digest moving image for a predetermined time in an infinite loop, displaying the beginning portion of the moving image as a still image, and the like can be considered. In this way, by displaying the digest moving image as a thumbnail on the LCD 146, the contents of the moving image can be easily confirmed with the thumbnail as well as the still image thumbnail.

  FIG. 10 is a flowchart illustrating processing of the moving image processing method in the imaging apparatus 100. First, in step S10, imaging of the operator by the in-camera is started. In the next step S12, face recognition is performed on the moving image captured by the in-camera, and the facial features are analyzed. In the next step S14, the facial expression is converted into a numerical value when the moving image is captured or the moving image is reproduced, and an evaluation value is calculated. In the next step S16, the digitized evaluation value is recorded on the same timeline as the moving image capturing or reproduction. In the next step S18, partial videos are sequentially extracted based on the evaluation value to generate a digest.

  As described above, according to the present embodiment, a desired digest video can be created based on the facial expression of the operator photographed by the in-camera. Therefore, it is possible to accurately extract the video of the part that the operator felt interested in when watching the video and the part that moved the heart, and it was possible to create the digest video as desired by the operator Become. As a result, efficient automatic editing of the moving image reflecting the operator's subjectivity becomes possible. Further, it is possible to automatically edit a moving image without depending on the state of a subject in the moving image (the presence / absence of a person, the orientation, the number of people, etc.) and the sound included in the moving image.

  Furthermore, if the operator recognizes in advance the rules for calculating the facial expression evaluation value, a digest video can be generated according to the operator's intention. Since the evaluation value can be calculated simultaneously with the moving image shooting, it is not necessary to re-decode the moving image data in order to calculate the feature amount, and the evaluation value calculation process can be performed efficiently. .

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 100 Imaging device 132,160 CMOS element 220 Face recognition part 222 Facial expression evaluation value calculation part 226 Movie editing part 228 Recording file production | generation part

Claims (18)

A first moving image imaging unit that images the operator;
An analysis unit for analyzing the characteristics of the operator's face from the first moving image captured by the first moving image capturing unit;
From the first moving image captured at the time of imaging or playback of the second moving image different from the first moving image, the facial expression of the operator is digitized based on the analysis result by the analysis unit, and the evaluation value is obtained. An evaluation value calculation unit for calculating,
A recording unit that records the numerical evaluation value on the same timeline as the second moving image;
A video editing unit that sequentially extracts partial videos of the second video based on the evaluation value and generates a digest;
A moving image processing apparatus comprising: A second moving image capturing unit that captures the second moving image;
The moving image processing apparatus according to claim 1, wherein the evaluation value calculating unit performs the numerical value when the second moving image capturing unit captures the second moving image.   The moving image processing apparatus according to claim 1, wherein the moving image editing unit sequentially extracts the partial moving images in a section in which the value of the evaluation value is larger than a predetermined threshold value.   The moving image according to claim 3, wherein the moving image editing unit extracts the partial moving image by changing the threshold value so that a total time of the partial moving image falls within a predetermined time. Image processing device.   The moving image editing unit sequentially extracts the partial moving images in a section obtained by adding a predetermined margin time before or after the section in which the value of the evaluation value is larger than a predetermined threshold value. The moving image processing apparatus according to claim 3.   The moving image processing apparatus according to claim 1, wherein the evaluation value calculation unit performs the digitization based on a predetermined rule.   The moving image processing apparatus according to claim 6, wherein the rule is defined such that weighting of an evaluation value is increased when a change in facial expression is severe.   The moving image processing apparatus according to claim 1, wherein the recording unit records the evaluation value as metadata in data of the second moving image.   The moving image processing apparatus according to claim 1, wherein the recording unit records the evaluation value as separate data that has a one-to-one correspondence with the data of the second moving image. Imaging the operator;
Analyzing the characteristics of the operator's face from the first moving image captured by the first moving image capturing unit;
Based on the analysis result of the facial feature, the facial expression of the operator is digitized and evaluated from the first video captured at the time of imaging or playback of the second video different from the first video Calculating a value;
Recording the digitized evaluation value on the same timeline as the second moving image;
Sequentially extracting partial videos of the second video based on the evaluation value to generate a digest;
A moving image processing method comprising: Capturing the second moving image,
The moving image processing method according to claim 10, wherein in the step of calculating the evaluation value, the quantification is performed when the second moving image is captured.   12. The moving image processing method according to claim 10 or 11, wherein in the step of generating the digest, the partial moving images in a section where the value of the evaluation value is larger than a predetermined threshold are sequentially extracted.   13. The partial moving image is extracted according to claim 12, wherein in the step of generating the digest, the partial moving image is extracted by changing the threshold value so that a total time of the partial moving image falls within a predetermined time. Video processing method.   In the step of generating the digest, sequentially extracting the partial moving images in a section obtained by adding a predetermined margin time before or after the section in which the value of the evaluation value is larger than a predetermined threshold value. The moving image processing method according to claim 12, wherein the moving image processing method is characterized.   15. The moving image processing method according to claim 10, wherein in the step of calculating the evaluation value, the digitization is performed based on a predetermined rule.   16. The moving image processing method according to claim 15, wherein the rule is defined such that weighting of an evaluation value is increased when a change in facial expression is severe.   The moving image processing method according to claim 10, wherein, in the recording step, the evaluation value is recorded as metadata in the data of the second moving image. The moving image processing method according to any one of claims 10 to 16, wherein, in the recording step, the evaluation value is recorded as separate data corresponding one-to-one with the data of the second moving image. .

Images