JP4639043B2 - Moving picture editing apparatus and moving picture editing method - Google Patents

Description

  The present invention relates to a moving image editing apparatus and a moving image editing method for editing captured moving image data, and in particular, a moving image editing apparatus capable of detecting a shooting failure section caused by a camera setting at the time of shooting and a camera operation error. And a moving image editing method.

  Conventionally, when shooting with a video camera, immediately before the video is recorded on the media, settings such as “camera shake”, “bokeh”, “overexposure / underexposure”, and “underexposure / overexposure” are automatically detected. However, many techniques have been proposed in which a mechanical, optical, or electronic setting change or suppression measure is taken in the camera so as to record an easy-to-see and good image on a medium.

  However, even if such a technique is used, depending on the actual shooting state, the above settings or shooting mistakes cannot be covered, and unintended videos are often recorded. On the other hand, skilled people and professional photographers tend to shoot manually without using the above technology so that the intention of photographing can be directly communicated to the camera. Or an unintended video may be recorded due to a shooting mistake. Therefore, these unintended shooting sections are cut or corrected by moving image editing work by a PC (Personal Computer) or a video editor after shooting.

  By the way, a moving image processing apparatus that automatically detects whether or not each section complies with a predetermined scene rule when a user detects a desired section from captured video data and edits the combined section is disclosed in the following patent. It is described in Document 1. The moving image processing apparatus described in Patent Document 1 acquires camera operation information at the time of shooting, and when editing a moving image, a panning end and start scene has both pan directions aligned. And a warning that informs the user that the video will be difficult to see if they do not meet these rules. To display.

  However, in order to automatically detect a predetermined scene rule, it is necessary to mount a mechanism for acquiring camera operation information on the camera side when camera operation information is acquired simultaneously with shooting and recorded together with a moving image. If this is not installed in the photographing unit, the user cannot obtain the benefit. In addition, camera operation information such as the camera settings and operation information described above is multiplexed and recorded in a moving image, but a separate mechanism for reading this camera operation information is also required on the video editing unit side. Both the video editing unit and the video editing unit become complicated.

  In addition, the mechanism for acquiring the camera operation information needs to work at the time of shooting in the shooting unit (camera). For this reason, real-time performance is required, and due to size restrictions including power on the shooting unit side. However, since it is difficult to mount such a powerful computing device, there is a problem in that it cannot have sufficient moving image analysis capability.

  Therefore, in Patent Document 1, when there is no means for obtaining camera operation information at the time of shooting, the transition between the clips (cut points) is first detected from the captured moving image using the inter-frame similarity, and detection is performed. A method is described in which camera operation is determined from a clip (frame group) that has been recorded using Hough transform or the like, and using these data, moving image editing is performed in the same manner as described above in accordance with predetermined rules.

JP 2003-69947 A

  However, in Patent Document 1, a motion vector vanishing point is obtained from a captured moving image using Hough transform, and the vanishing point that has obtained the maximum number of votes among a plurality of vanishing points is used as a vanishing point with respect to the background. The camera operation is determined by solving the constraint formula related to the operation information. Predetermined by an operation error or shooting error such as “shake,” “out of focus,” “overexposure / underexposure”, “underexposure / overexposure”, etc. Cannot be detected.

  As described above, the video data may include unnecessary shooting sections that are not intended by the photographer, particularly when editing moving image data for a long time, which may be caused by an operation error or a shooting error. It is very time consuming to find a shooting section that is difficult to see or intended by the user.

  The present invention has been proposed in view of such a conventional situation, and for moving image data that has already been recorded, a section that is highly likely to be deleted or corrected during editing is detected by moving image analysis, An object of the present invention is to provide a moving image editing apparatus and a moving image editing method capable of cutting or correcting a section at high speed and easily by displaying the information together with moving image data during editing.

  In order to achieve the above-described object, a moving image editing apparatus according to the present invention includes a storage unit that stores moving image data shot by a shooting unit, and analyzes the moving image data to obtain a predetermined reproduction standard. A plurality of determination items for determining whether or not the condition is satisfied are evaluated individually for each frame or field based on at least one of the luminance distribution, luminance differential value distribution, and motion vector distribution for each frame or field of the moving image data. Based on the image analysis means for calculating the value, and the number of shooting failure frames or the number of fields in which the evaluation value for each of the determination items calculated by the image analysis means is less than a predetermined threshold that satisfies the reproduction criterion, From the moving image data, a shooting failure candidate section determined not to satisfy the reproduction criterion is detected for each determination item, and Candidate section detecting means for detecting a section in which at least one shooting failure candidate section for each fixed item is detected as a recommended cut section, and the evaluation value for each determination item on the time axis of the moving image In addition to displaying a graph in units, on the time axis, display means for displaying the shooting failure candidate section for each determination item and the recommended cut section together with the moving image data, and deleting or correcting the moving image data And learning means for resetting the reproduction reference for detecting a shooting failure candidate section based on the evaluation value of the moving image data deleted or modified by the operation means.

  In the present invention, a shooting failure candidate section that is determined not to satisfy a predetermined reproduction criterion is automatically detected and displayed together with moving image data. Editing can be performed while checking the video of the section.

  Further, the image analysis means individually calculates evaluation values for a plurality of determination items for determining whether or not the reproduction criterion is satisfied, and the candidate section detection means is based on the evaluation values for each determination item. The above shooting failure candidate sections can be detected, a plurality of determination items are set, and the shooting failure candidate sections are individually detected based on the evaluation values in the respective determination items, so that the sections according to the type of video and the editing purpose thereof. You can select to delete or modify.

  Furthermore, the display means can display the shooting failure candidate section on the time axis of the moving image, thereby further improving the efficiency of editing work.

  Furthermore, the candidate section detecting means detects a recommended cut section based on each shooting failure candidate section detected for each determination item, and the display means displays the recommended cut section together with the moving image data. By detecting the shooting failure candidate sections in the plurality of determination items, the user can be made to recognize the recommended cut section having a high certainty as the section to be deleted.

  The image analysis means can calculate the evaluation value in units of frames or fields based on at least one of a luminance distribution, a luminance differential value distribution, and a motion vector distribution for each frame or field of the moving image data. Evaluation items can be calculated by providing judgment items for evaluating whether or not “camera shake”, “out-of-focus”, “out-of-white / black-out”, “under-exposure / over-exposure”, and the like have occurred.

  Further, the candidate section detection means may detect, as the shooting failure candidate section, a section in which a shooting failure frame or field whose evaluation value is less than a predetermined threshold that satisfies the reproduction criterion is continuous for a predetermined section or more. It is determined that it is a shooting failure frame or field because it is not clear, the whole has moved greatly, the brightness is not appropriate, whiteout, blackout, etc. occur, and it is continuous Can be detected as a shooting failure candidate section.

  Still further, the candidate section detecting means is configured such that a plurality of sections in which the intervals at which shooting failure frames or fields whose evaluation values satisfy the reproduction criterion are less than a predetermined threshold appear are less than the predetermined intervals are consecutive. If the interval exceeds a predetermined interval, the interval can be detected as the above-mentioned shooting failure candidate interval, and shooting failure occurs when an evaluation value drops at a predetermined interval due to camera shake or the like, or a shooting failure frame or field appears. It can be detected as a candidate section.

  Further, the display means can display the evaluation value for each of the determination items, and perform editing work with reference to the evaluation value as an editing index even in a frame that is not detected as a shooting failure candidate section. Can do.

  Further, the reproduction criterion for detecting the shooting failure candidate section based on the evaluation value of the moving image data deleted or modified by the operation unit, further comprising an operation unit for deleting or correcting the moving image data. The learning means for resetting can be provided, and the accuracy of detection of the shooting failure candidate section can be improved by learning.

  The moving image editing method according to the present invention includes a storage step of storing moving image data shot by a shooting unit, and a plurality of determinations whether the moving image data is analyzed to satisfy a predetermined reproduction criterion. An image analysis step for calculating the evaluation value individually for each frame or field based on at least one of the luminance distribution for each frame or field of the moving image data, the luminance differential value distribution, and the motion vector distribution for the determination item; Based on the number of unsuccessful frames or the number of fields in which the evaluation value for each of the determination items calculated in the image analysis step is less than a predetermined threshold that satisfies the reproduction criterion, the reproduction from the moving image data is performed. A shooting failure candidate section determined not to satisfy the criterion is detected for each determination item, and the shooting failure candidate section for each determination item is A candidate section detection step for detecting at least one detected section as a recommended cut section, and the evaluation value for each determination item on the time axis of the moving image is displayed as a graph in units of frames, and the time On the axis, a display step for displaying the shooting failure candidate interval for each determination item and the recommended cut interval together with the moving image data, an operation step for deleting or correcting the moving image data, and the operation And a learning step of resetting the reproduction reference for detecting a shooting failure candidate section based on the evaluation value of the moving image data deleted or corrected in the step.

  According to the moving image editing apparatus and method of the present invention, the moving image data shot by the shooting unit is stored, the moving image data is analyzed, and the moving image data is determined in advance from the moving image data based on the analysis result. Since the shooting failure candidate section determined to not satisfy the reproduction criterion is detected and the shooting failure candidate section is displayed together with the moving image data, the user who edits the moving image does not satisfy the predetermined reproduction criterion. The editing work can be performed while checking the video of the shooting failure candidate section determined to be, and the trouble of searching for the shooting failure candidate section can be saved, the editing work becomes highly efficient, and the shooting failure candidate section is automatically selected. Searching can prevent oversight.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In this embodiment, the present invention can be applied to “camera shake”, “blurring”, “overexposure / blackout”, “underexposure / The present invention is applied to a moving image editing apparatus and method that automatically detects a shooting failure section such as “over” to improve the efficiency of the moving image editing work and prevent the shooting failure section from being overlooked.

  FIG. 1 is a block diagram showing a moving image editing system according to an embodiment of the present invention. As shown in FIG. 1, a moving image editing system 1 includes a video camera 10 for capturing an image, and an information processing apparatus such as a PC for recording the captured moving image data and editing the moving image data. 20.

  The video camera 10 includes a CCD (Charge Coupled Devise) that converts an image into an electric signal, a recording signal processing module 12 that converts an electric signal from the CCD into a predetermined recording signal, and a media input / output (I / O) module 13. And a recording medium 14 on which moving image data is recorded. Further, a serial interface input / output module (IEEE (Institute of Electrical and Electronics Engineers)) for reading moving image data recorded on the recording medium 14 from the media I / O module and outputting the moving image data to an external device. 1394 I / O module) 15. The recording medium 14 is a magnetic tape, an optical disk, a magneto-optical disk, a magnetic disk, a nonvolatile memory, or the like. The video camera 10 is connected to the information processing apparatus 20 via, for example, IEEE 1394, so that moving image data recorded on the recording medium 14 can be output to the information processing apparatus 20.

  The information processing device 20 is read from the IEEE 1394 I / FCard 21 that receives the moving image data output from the video camera 10, the hard disk HD (HardDisk) 22 that records the moving image data received by the IEEE 1394 I / FCard 21, and the HD 22. A random access memory (RAM) 23 for storing the program and a CPU 24 for executing various processes in accordance with the program loaded in the RAM 23. The IEEE 1394 I / FCard 21, the HD 22, the RAM 23, and the CPU 24 are connected via a bus 25. ing.

  A video input / output unit (Video I / O) 26 is also connected to the bus 25, and is connected to a display 29 via a digital video interface (VDI) 28. In addition, a USB (Universal Serial Bus) I / F 27 is connected to the bus 25, and is connected to a keyboard 31 and a mouse 32 via the USB 29.

  The display 29 can display a GUI (Graphic User Interface) for browsing moving image data recorded on the HD 22. Further, as will be described later, in the moving image editing system 1, for example, “camera shake”, “out-of-focus”, “overexposure / underexposure”, “underexposure / overexposure”, etc. occur due to camera setting mistakes or shooting mistakes. It is possible to automatically detect a shooting failure section (hereinafter referred to as an NG candidate section) that is difficult to see or unintended by the photographer, and display this by the GUI when editing a moving image.

  The user can edit the moving image by playing back the recorded moving image data and correcting or deleting the NG candidate section in the moving image data detected by the moving image editing system 1 using this GUI. it can. The GUI displayed on the display 29 can be operated by a keyboard 31 and a mouse 32 which are input devices from the user.

  First, an operation flow in the moving image editing apparatus according to the present embodiment will be described. FIG. 2 is a diagram showing a workflow until an NG candidate section is detected and displayed on the GUI by the moving image editing apparatus 1 using the system of FIG.

  First, a moving image (video) is captured by the CCD 11 of the video camera 10, and moving image data is captured and recorded on the recording medium 14 via the recording signal processing module 12 and the media I / O module 13. Then, the moving image data recorded on the recording medium 14 is read by the media I / O module 13, received by the IEEE 1394 I / F card 21 via the IEEE 1394 I / O module, and recorded on the HD 22 (step S1). .

  Next, image analysis of the moving image data recorded on the HD 22 is performed. In the image analysis, the moving image data of the HD 22 is read into the RAM 23, and the CPU 24 calculates the luminance distribution, the luminance differential value, the motion vector distribution, etc. for each frame by the method described later (step S2).

  Next, based on the obtained image analysis result, whether the visibility of the moving image data is “sharp and clear”, “whether there is little movement”, “whether there is little change in movement”, Each judgment item that determines whether or not it meets the specified playback criteria, such as whether it is "too bright or too dark" or "no whiteout or blackout", In addition, if there is a section longer than a certain shooting section or a section that does not satisfy the standard, it is extracted and stored as an NG candidate section (step S3).

  For example, when the user edits a moving image or when the processing of step S2 and step S3 ends, the NG candidate interval is displayed on the GUI time axis together with the moving image, thereby deleting the NG candidate interval, This is to help the user to correct the moving image editing work (step S4).

  Further, the processing in step S2 and step S3 may be performed after the processing in step S1 for recording moving image data on the HD 22 is completed, and is performed in a pipeline in parallel with the processing in step S1. May be.

  Here, in the moving image editing system according to the present embodiment, moving image data is analyzed by moving image data analysis processing in step S2, NG candidate section detection processing in step S3, and NG candidate section deletion / correction processing in step S4. The NG candidate section and the recommended cut section selected based on each NG candidate section are notified to the user of shooting sections that are considered difficult to see, making it easy to eliminate, thereby making editing work highly efficient It is.

  First, the moving image analysis process in step S2 will be described in detail. Here, as evaluation of the visibility of the moving image data, A: “sharp and clear (Dis)”, B: “no movement (St1)”, C: “change in movement” 5: Ease of visibility: whether it is “low (Stb)”, D: “does not be too bright or too dark (Lum)”, E: “does not have many overexposure or blackout (Sat)” An example in which evaluation is performed on the determination items will be described. It should be noted that all the evaluation values indicate that the higher the value is, the easier it is to see the moving image data, that is, it is estimated that the shooting is successful.

  Here, the item for determining whether or not “there is little movement” is evaluated as being highly evaluated as there is little movement between frames. This is because, for example, a subject can hear a sound from behind while photographing the front, and the photographer moves the camera forward. For example, when moving from the rear to the rear, the movement between the moving frames increases, and the evaluation value of “less movement” decreases. In such a case, when the moving image is cut and it is desired to use the moved image, the determination item of whether “there is little movement” is effective.

  In addition, in the determination item of whether or not “the change in motion is small”, the higher the evaluation is the smaller the change in motion, this is mainly to detect camera shake. Since the evaluation value decreases due to an increase in the change in camera shake, it is possible to detect a camera shake section.

  In the present embodiment, the following eight parameters are calculated for each frame in the moving image data as parameters for calculating an evaluation value indicating the evaluation of each determination item for visibility.

(1) Average luminance (2) Luminance standard deviation (3) Whiteout ratio (4) Black crush rate (5) Average contour difference (6) Pattern matching success rate P
(7) Average movement amount Avg_m
(8) Average moving direction Avg_d

  These parameters are used to calculate an evaluation value indicating the evaluation of each visibility evaluation item as follows.

A: Evaluation of “sharp and clear” (2) Luminance standard deviation (5) Average contour difference B: Evaluation of “no movement” (6) Pattern matching success rate (7) Average movement amount (8) Average moving direction C: “There is little change in movement”
(7) Average moving amount (8) Average moving direction D: Evaluation of whether it is “too bright or too dark” (1) Average luminance E: Evaluation of whether there are not many overexposures or blackouts (3) White skip ratio (4) Black crush ratio

  FIG. 3 is a flowchart showing a moving image data analysis processing method for calculating (1) to (8) described above. As shown in FIG. 3, first, the input moving image data is decompressed (step S11), and the luminance distribution is calculated for each frame (step S12).

In the luminance distribution calculation in step S12, a luminance histogram is obtained for a certain frame in the moving image data, and (1) average luminance, (2) luminance standard deviation, (3) whiteout rate, and (4) black crushing rate are obtained. . FIG. 4 is a graph showing a luminance histogram in one frame, with the horizontal axis representing brightness and the vertical axis representing the number of pixels. From this luminance histogram, the average value of the luminance histogram is calculated as (1) average luminance, and the standard deviation of the luminance histogram is calculated as (2) luminance standard deviation. Further, (3) overexposure ratio, and (4) black crushing rate is easily calculated by a formula shown below from the number of pixels Y _min with a minimum value of the pixel number Y _max and luminosity with maximum brightness be able to.

(1) Average luminance = Average value Y _{avg of} luminance histogram
(2) Luminance standard deviation = Standard deviation of luminance histogram (3) Whiteout ratio (%) = (number of pixels having the maximum value of brightness Y _max / total number of pixels) × 100
(4) Black crushing rate (%) = (number of pixels having the minimum value of brightness Y _min / total number of pixels) × 100

  Returning to FIG. 3, next, the luminance differential value distribution calculation is performed (step S13). Thus, (5) an average contour difference is obtained. FIG. 5 is a graph showing a differential value histogram used in the process of calculating the luminance differential value distribution, with the luminance differential value on the horizontal axis and the number of pixels on the vertical axis. The luminance differential value shown in FIG. 5 takes the difference between adjacent pixels in the horizontal and vertical directions of each pixel for a certain frame in the moving image data, and the absolute value of the difference between adjacent pixels in the horizontal and vertical directions is This is an example in which a histogram is obtained using a larger value as a differential value. Then, for pixels having a differential value equal to or greater than a certain threshold Th, that is, for pixels included in the region P in FIG. 5, the average value of the luminance differential values included in the region P is obtained when the number of pixels reaches a certain amount or more. Obtain (5) average contour difference shown below. Here, when it does not reach a certain amount or more, the threshold Th that is the minimum value is set as (5) average contour difference. Here, the threshold value Th is a pixel that forms a sharp portion of the frame when the pixel has a larger luminance differential value than the threshold value Th, and a pixel that forms a blurred portion of the frame when the pixel has a smaller luminance differential value than the threshold value Th. It is a value set as appropriate so that it can be considered.

(5) Average contour difference = average differential value of pixels having a differential value greater than or equal to the threshold Th

  Next, as shown in step S14 of FIG. 3, the motion vector distribution is calculated to determine (6) pattern matching success rate, (7) average movement amount, and (8) average movement direction.

  6 and 7 show graphs used in the motion vector calculation process. In this example, each block (for example, 16 × 16 pixels) in a certain frame (current frame) is subjected to pattern matching with the previous frame, and its movement amount (movement distance) and movement direction are obtained to form a histogram. That is, FIG. 6 is a graph showing a histogram of the movement amount of each block with the movement amount on the horizontal axis and the number of blocks on the vertical axis, and FIG. 7 shows the movement direction on the horizontal axis and the block on the vertical axis. It is a graph which shows the histogram of the moving direction of each block taking a number. When the moving image data is an interlaced image, pattern matching may be performed between the odd and even fields instead of the previous frame. In the pattern matching used here, simply the absolute value of the luminance differential between the target block (target block) of the current frame and the block of the same size as the target block in the vicinity of the target block coordinates in the previous frame. A block with the smallest sum of the values is searched, and if this distance is less than or equal to the threshold, pattern matching is successful for that block. Thereby, (6) the pattern matching success rate can be easily obtained, but this success rate is related to the range in which the search is performed. The “movement” to be obtained here is the movement of the subject, and this is defined as a “small movement” block rather than a “large movement” block, so this search range does not have to be so wide. That is, a block having a large movement that does not succeed in matching within a certain search range is not used for the evaluation in the “less movement” determination item. The average values avg_m and avg_θ of the movement amount histogram shown in FIG. 6 and the movement direction histogram shown in FIG. 7 are (7) average movement amount and (8) average movement direction, respectively.

(6) Pattern matching success rate (%) = (number of successful matching blocks / total number of blocks) × 100
(7) Average movement amount = Average value of movement amount avg_m
(8) Average moving direction = Average value avg_θ in the moving direction

  After calculating the motion vector distribution in this way, it is determined whether or not it is the last frame (step S15). If it is the last frame, the image analysis processing is terminated. On the other hand, if it is not the last frame, the next frame is read (step S16), and the processing from step S11 to step S15 is repeated. As described above, the parameters (1) to (8) necessary for calculating the evaluation value of each visibility determination item are obtained for each frame.

  Next, the NG candidate section detection process in step S3 shown in FIG. 2 will be described. First, in order to evaluate the visibility of moving image data using the parameters (1) to (8) obtained in steps S12 to S14 shown in FIG. 3, A: “Sharp and clear (Dis) ) ”, B:“ There is little movement (Stl) ”, C:“ There is little change in movement (Stb) ”, D:“ It is neither too bright nor too dark (Lum) ” Whether or not E: “Evaluation value in each determination item of whether there are not many overexposures or blackouts (Sat)” is calculated. In the following example, all evaluation values are calculated as values from 0 to 100 for simplicity of explanation.

The evaluation value (Dis) of the determination item A indicating whether or not a certain frame is “sharp and clear” is obtained in step S12 of FIG. 3 (2) luminance standard deviation (Def) and obtained in step S13 (5 ) It can be calculated using the average contour difference (Edg). First, a “sharp and clear” reference image 1 and a “blurred and unclear” reference image 2 are prepared in advance. For these reference images 1 and 2, (2) luminance standard deviation and (5) average contour level. Calculate the difference. Then, the luminance standard deviation of the reference image 1 and the reference image 2 is set as the upper limit value and the lower limit value (Def _max , Def _min ) of the luminance standard deviation, respectively, and the average contour difference of the reference image 1 and the reference image 2 is determined as the average contour difference. Are the upper limit value and the lower limit value (Edg _max , Edg _min ), respectively. Then, (2) luminance standard deviation in the evaluation value calculation target frame, (5) average contour difference, and (Def _max , Def _min ) and (Edg _max , Edg _min ) obtained in advance by the reference images 1 and 2. ) To calculate the respective ratios as shown in the following formula (1), and after performing appropriate weighting (W _def , W _edg ) so that the evaluation value (Dis) is 0 to 100, The sum is taken and this is used as the evaluation value of the judgment item A for judging “sharp and clear”. That is, the evaluation value (Dis) is calculated by the following formula (1). A higher evaluation value (Dis) indicates a sharper and clearer image.

Further, the evaluation value (Stl) in the determination item B indicating whether or not a certain frame is “little movement” is obtained by (6) pattern matching success rate (Pm) obtained in step S14 of FIG. It can be calculated by using the movement amount (Dis _avg ) and (8) the average movement direction (Dir _avg ). Here, (6) the pattern matching success rate indicates whether or not “subject is captured”, and the average movement vector and the average movement vector obtained in (7) and (8) are “subject”. It is assumed that it shows how much is moving. That is, in a completely still image, (6) the pattern matching success rate is 100, and the average movement vector obtained in (7) and (8) is (0, 0). Here, “less motion” is calculated. As a reference, 100 is a case where a stationary subject is photographed while still, and a case where the camera is panned at a limit speed capable of pattern matching. 50, the case where all are large movements that cannot be pattern matched is defined as 0. Based on this, the maximum distance Dis _max that succeeds in pattern matching is obtained, and this “little motion”, that is, the evaluation value (St1) of the determination item B as to whether or not the subject is successfully captured is expressed by the following equation: Calculate from 0 to 100 according to (2).

  Also, the evaluation value (Stb) of the determination item C indicating whether or not “motion change is small” for detecting the “hand shake” section is calculated from the average movement vector obtained in (7) and (8). can do. First, the difference between the average movement vector in the current frame and the average movement vector in the previous frame obtained in (7) and (8) is obtained, and the speed change amount is obtained based on this vector distance. When the average movement vector of the current frame I is (uI, vI) and the average movement vector of the previous frame (I-1) is (uI-1, vI-1), a determination item of “little change in motion” The evaluation value (Stb) in C is calculated from 0 to 100 by the following equation (3).

In addition, the evaluation value (Lum) of the determination item D indicating whether it is “not too bright or too dark” is the (1) average luminance (Lum _avg ) and the maximum luminance value obtained in step S11 of FIG. Lum _max is used, and can be calculated as a value of 0 to 100 by the following formula (4).

Next, the evaluation value (Sat) of the determination item E indicating whether or not “there is no whiteout or blackout” is obtained in step S12 of FIG. 3 (3) whiteout rate (Sat _max ), ( 4) Using the black crushing rate (Sat _min ), it can be calculated as a value of 0 to 100 by the following formula (5).

  Thus, evaluation values for each of the five determination items in each frame in the moving image data are obtained. And using these, the detection process of the NG candidate area of step S3 shown in FIG. 2 is performed. This NG candidate section detection process is performed for each determination item, that is, whether A: “sharp and clear (Dis)”, B: “less movement (St1)”, or C: “change in movement”. 5 judgment items: whether it is “low (Stb)”, D: “not too bright or too dark (Lum)”, E: “does not cause excessive whiteout or blackout (Sat)” NG candidate sections are individually detected.

  First, a method for detecting an NG candidate section for a determination item other than the determination item C indicating “less change in motion (Stb)” will be described. For these detections, each item is classified into an OK frame or an NG frame by determining the threshold value of the evaluation value obtained for each frame. For example, an NG frame having a minimum section length n of about n = 10 to 20, for example. Are detected as NG candidate sections.

  FIG. 8 is a flowchart showing an NG candidate section detection processing method for determination items other than E: “less change in motion (Stb)”, that is, the determination items A, B, D, and E. When each evaluation value of each frame is calculated as described above, the NG candidate section detection process is started. Here, a method for detecting a section having a small evaluation value as an NG candidate section in determination item A: “sharp and clear (Dis)” will be described as an example. First, as shown in FIG. 8, it is determined whether or not the current frame (frame count) i that is sequentially input in time series is equal to or less than the last frame I (step S21), and the process proceeds to step S22 until the last frame I. In step S22, it is determined whether or not the frame is an NG frame depending on whether or not the evaluation value (Dis) of the equation (1) is less than a predetermined threshold. If it is determined that the frame is not an NG frame, the process proceeds to the next frame (step S23), and the process from step S21 is repeated.

  If it is determined in step S22 that the threshold is determined to be sharp and not clear and an NG frame, the determined frame is set as a temporary NG candidate section temporary start / end frame (step S24). Then, the next frame i is read (step S25), it is determined whether or not this is larger than the final frame I (step S26), and the process proceeds to step S27 up to the final frame I. Similarly to step S22, the above equation (1) Whether or not the frame is an NG frame is determined based on whether or not the evaluation value (Dis) is less than a predetermined threshold. If this frame is an NG frame, the frame is set as a temporary end frame of the temporary NG candidate section (step S28). Then, the processing from step S25 for reading the next frame is repeated.

  On the other hand, if it is determined in step S26 that there is no next frame, and if it is determined in step S27 that the current frame i is not an NG frame, temporary NG that is a continuous section that has been determined to be an NG frame until then. It is determined whether or not the length of the candidate section (the number of frames) is not less than the minimum section length n (step S29). If the length of the temporary NG candidate section is less than n, it is too short to be detected as an NG candidate section, so this temporary NG candidate section is cleared (step S31), and the processing from step S21 is repeated. If the length of the temporary NG candidate section is greater than or equal to n in step S29, the temporary NG candidate section is stored as an NG candidate section in the HD 22 or the like (step S30), and the process proceeds to step S31 to store the stored temporary NG candidate section. The NG candidate section is cleared and the processing from step S21 is repeated. Note that the HD 22 shown in FIG. 1 is suitable as the location for storing the NG candidate section in step S30, but it can also be stored in another random accessible memory or an optical disk.

As a result, as shown in FIG. 9A, a section in which the NG frame F _NG continues for n frames or more is detected as the NG candidate section I _c . In FIG. 9, for the convenience of drawing, when n = 5 or more, it is detected as an NG candidate section I _c .

In addition, regarding the continuous determination of NG candidate sections, in a continuous section, several OK frames are overlooked to give a wider range of determination, and detection of a portion that is difficult to see due to a failure in photographing or the like is detected. It is also possible to prevent leakage. Specifically, in FIG. 9A, for example, the third frame from the left is an OK frame F _OK (1), but this is regarded as an NG frame, and the start frame of the NG candidate section I _c is defined as F. It is possible to detect not as _NG (3) but as NG candidate section I _c ′ as F _NG (1).

  Further, in FIG. 8, the process of detecting the determination item A: the section that is not “sharp and clear (Dis)” as an NG candidate section has been described as an example, but the determination item B: “motion is small (Stl)” cannot be evaluated. Section, judgment item C: a section that cannot be evaluated as “not too bright or too dark (Lum)”, and judgment item D: a section that cannot be evaluated as “not overexposed or underexposed (Sat)” as NG section candidates The detection method can be similarly performed. That is, in step S22 and step S27 for determining whether or not the frame is an NG frame, the threshold value may be determined using the evaluation values shown in the equations (2), (4), and (5), respectively.

Next, calculation of the NG candidate section for the determination item E: “little change in motion (Stb)” will be described. Also in this calculation, it is determined whether each frame is an OK frame or an NG frame simply by threshold determination. In this case, the maximum interval (maximum number of frames) at which an NG frame appears is m (< A section in which the sections within n) continue for the minimum section length n or more is defined as an NG candidate section. That is, as in FIG. 9A, when n = 5, as shown in FIG. 9B, for example, the interval between the NG frame F _NG (2) and the NG frame F _NG (3) is 4 frames. Yes, m (= 4) <n = 5 is satisfied. The interval between the NG frame F _NG (3) and the NG frame F _NG (4) is two frames, and similarly satisfies m (= 2) <n = 5. Further, it is assumed that the frames after the NG frame F _NG (5) are OK frames continuously for 5 frames or more. In this case, an NG frame F _NG (1) to _NG frame F _NG (5) is detected as an NG candidate section for “little change in motion (Stb)”, and the length of this section is 11 frames. since a minimum interval length n = 5 or more, it detects this as NG candidate sections _{I c.}

  This is because the evaluation values other than “less change in motion (Stb)” are NG continuously (evaluation becomes lower), but there is a shake about “less change in motion (Stb)”. This is because NG is expected to occur, so that it does not become NG continuously but becomes NG at a certain interval. As described above, the detection method of the NG candidate section is appropriately set based on the cause of the NG frame.

  And finally, according to step S4 of FIG. 2, as the processing results of step S1 to step S3 of FIG. 2, the motion image data, the visibility of the image obtained based on the evaluation value, and the NG candidate section are estimated thereby. The recommended cut section is displayed to the user on the GUI of the display 29 in FIG. Next, this GUI will be described.

  FIG. 10 is a schematic diagram showing a typical GUI displayed on the display when the user edits a moving image. The GUI 50 includes a monitor 51 that displays a moving image, a scene list 52 that displays a list of each scene of moving image data, evaluation values of image visibility calculated in step S2 of FIG. 2 in time order, and NG candidates. A timeline 53 for displaying sections and the like is included.

  The monitor 51 is an area for reproducing the moving image data file selected in the scene list 52 and confirming the actual video. The scene list 52 is an area for displaying a list of moving image data files recorded on the HD 22 in the PC 20 shown in FIG. In this system, the moving image data to be recorded on the HD 22 shown in FIG. 1 is divided into predetermined sections based on ON / OFF information of the recording button on the camera side, detection of scene change points, user operation, etc., for example, as a file I'm organizing.

  The timeline 53 is an area for displaying the evaluation value, the NG candidate section, and the like calculated by the above-described method for the moving image data file selected in the scene list 52. Here, the horizontal axis is the time direction, and in addition to the columns for displaying the “time code” and “video” referenced in the vertical direction, the evaluation values of the visibility items calculated by the above method are edited. Displayed side by side as an index. Each evaluation value is displayed in a graph in units of frames on the vertical axis in a row indicating each determination item, thereby enabling editing in units of frames.

  Furthermore, it is assumed that a display is made so that the user can recognize the NG candidate section detected in step S3. Specifically, the NG candidate section may be displayed with a frame, or the NG candidate section and the other section may be displayed in different colors.

  Here, in the present embodiment, visibility determination item A calculated in step S2 of FIG. 2: whether or not it is “sharp and clear (Dis)”, determination item B: “motion is small (Stl ) ”, Judgment item C:“ There is little change in motion (Stb) ”, Judgment item D:“ Never too bright or too dark (Lum) ”, Judgment item E:“ White ” Each evaluation value of whether or not there is not much skipping or black crushing (Sat) is subtracted from 100, and each of “movement”, “camera shake”, “blur”, “flying / crushing”, “brightness” It is assumed that the evaluation item is a negative evaluation value for the visibility item and is displayed as an editing index. That is, in order to facilitate deletion / correction of the NG candidate section, an evaluation value graph opposite to the evaluation value of visibility is displayed as an index (evaluation value) of the visibility of each judgment item of visibility. The value obtained by subtracting the visibility value calculated as described above from the maximum value of 100 is displayed as the visibility evaluation value graph. This makes it easy for the user to visually recognize and correct or delete a shooting section that is difficult to see in the captured image or is not intended by the photographer.

  Further, in the timeline 53, NG candidate sections are individually displayed for the determination items indicating the difficulty of seeing, and further, taking into account the user's operability, a section obtained by ORing them is displayed as a recommended cut section. To do.

  Then, the user selects, for example, the recommended cut section detected by the moving image editing system 1 according to the present embodiment using the keyboard 31 or the mouse 32 shown in FIG. Deletion / correction is performed for it. Since each evaluation value is displayed in a graph, it is possible to make changes such as extension / reduction of a section to be deleted / modified using this as a guide.

  In addition, since the NG candidate section is displayed so as to be visible for each of the difficulty-of-visibility determination items, even if it is a cut recommended section, if there are few determination items that are NG candidate sections, deletion / Priority of editing with reference to the NG candidate section of each judgment item, such as no modification or deletion / correction in the order of the number of judgment items that are NG candidate sections in the recommended cut section It is possible to determine whether or not to edit, etc., and editing in a short time is also possible.

  Furthermore, even if it is a section that is not displayed as a recommended cut section (NG candidate section), for example, a short section with a small number of NG frames to be detected as an NG candidate section, it is difficult to see in the visibility item. Since a section having a high evaluation value can be visually recognized, it can be deleted / corrected as required by the user. That is, the user can make corrections / deletions for each evaluation value, each NG candidate section, and each recommended cut section indicated as an index of difficulty of viewing according to the editing purpose, editing time, and the like.

  Furthermore, statistics on the reference evaluation value can be obtained by deleting or correcting this NG candidate section, or by providing feedback to the NG candidate section, for example, by selecting whether to add the evaluation value of this section to the reference video. By such learning work, the threshold value for discriminating the OK frame / NG frame of each frame is appropriately changed, or the minimum interval length n or the maximum interval m detected as the NG candidate interval is appropriately changed. Further, it is possible to perform NG candidate section detection in accordance with the user's intention.

  As described above, according to the moving image editing system of the present embodiment, a video including a shooting failure section such as “blur”, “blur”, “overexposure / blackout”, “underexposure / overexposure”, and the like. Thus, by automatically searching for NG candidate sections, it is possible to save the trouble of shooting due to an operation mistake or setting mistake or to search for a shooting section that is not intended by the user, and to improve the efficiency of the editing work. By automatically detecting the section, it is possible to prevent overlooking of editing.

It is a block diagram which shows the moving image editing system in embodiment of this invention. It is a figure which shows a workflow until it detects an NG candidate area using a moving image editing system, displays it on GUI, and performs an edit operation. It is a flowchart which shows the analysis processing method of the moving image data which calculates each parameter for calculating | requiring each evaluation value and evaluation value. It is a graph which shows the brightness | luminance histogram in 1 frame, taking lightness on a horizontal axis and taking the number of pixels on a vertical axis | shaft. It is a graph which shows a differential value histogram used in the process of luminance differential value distribution calculation, taking a luminance differential value on the horizontal axis and taking the number of pixels on the vertical axis. It is a graph which shows the histogram of the movement amount of each block, taking a movement amount on a horizontal axis and taking the number of blocks on a vertical axis. It is a graph which shows the histogram of the moving direction of each block, taking a moving direction on the horizontal axis and taking the number of blocks on the vertical axis. It is a flowchart which shows the NG candidate area detection processing method in determination item A, B, D, E. (A) is a diagram showing judgment items A, B, D, the NG candidate section _{I c} in E, illustrates the NG candidate section _{I c} in (b) is determined Item C. It is a schematic diagram which shows the typical GUI displayed on a display when a user edits a moving image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Video camera, 12 Recording signal processing module, 13 Media input / output module, 14 Recording medium, 15 Serial interface input / output module, 20 Information processing apparatus, 21 IEEE1394I / FCard, 25 Bus, 26 Video input / output part, 27 USB I / F , 29 Display, 31 Keyboard, 32 Mouse, 51 Monitor, 52 Scene list, 53 Timeline, 54 User operation

Claims (6)

Storage means for storing moving image data photographed by the photographing means;
A plurality of determination items for analyzing the moving image data to determine whether or not a predetermined reproduction criterion is satisfied, a luminance distribution, a luminance differential value distribution, and a motion vector distribution for each frame or field of the moving image data. Image analysis means for individually calculating the evaluation value for each frame or field based on at least one of
Based on the number of unsuccessful shooting frames or the number of fields in which the evaluation value for each of the determination items calculated by the image analysis means is less than a predetermined threshold that satisfies the reproduction standard, the reproduction standard is obtained from the moving image data. Candidate section detection means for detecting a shooting failure candidate section determined not to satisfy each of the determination items and detecting a section in which at least one shooting failure candidate section for each determination item is detected as a recommended cut section When,
On the time axis of the moving image, the evaluation value for each determination item is displayed in a graph in units of frames, and on the time axis, the shooting failure candidate interval for each determination item, and the recommended cut interval Displaying means together with the moving image data;
Operating means for deleting or correcting the moving image data;
A moving image editing apparatus comprising: learning means for resetting the reproduction criterion for detecting a shooting failure candidate section based on the evaluation value of the moving image data deleted or modified by the operation means.   2. The candidate section detection means detects, as the shooting failure candidate section, a section in which a shooting failure frame or field in which the evaluation value is less than a predetermined threshold value that satisfies the reproduction criterion is continued for a predetermined section or more. Video editing device.   The candidate section detecting means is a predetermined section in which a plurality of sections in which the shooting failure frame or field in which the evaluation value is less than a predetermined threshold that is considered to satisfy the reproduction criterion appear is less than the predetermined interval. The moving image editing apparatus according to claim 1, wherein, when the number of times becomes above, the section is detected as the shooting failure candidate section. A storage step for storing moving image data photographed by the photographing means;
A plurality of determination items for analyzing the moving image data to determine whether or not a predetermined reproduction criterion is satisfied, a luminance distribution, a luminance differential value distribution, and a motion vector distribution for each frame or field of the moving image data. An image analysis step for calculating the evaluation value individually for each frame or field based on at least one of
Based on the number of unsuccessful frames or the number of fields in which the evaluation value for each of the determination items calculated in the image analysis step is less than a predetermined threshold that satisfies the reproduction criterion, the reproduction from the moving image data is performed. Candidate section detection for detecting a shooting failure candidate section determined not to satisfy the criterion for each determination item and detecting a section in which at least one shooting failure candidate section for each determination item is detected as a recommended cut section Process,
On the time axis of the moving image, the evaluation value for each determination item is displayed in a graph in units of frames, and on the time axis, the shooting failure candidate interval for each determination item, and the recommended cut interval Displaying with the moving image data,
An operation step for deleting or correcting the moving image data;
And a learning step of resetting the reproduction reference for detecting a shooting failure candidate section based on the evaluation value of the moving image data deleted or modified in the operation step.   The said candidate area detection process detects the area where the said frame or field in which the said evaluation value is less than the predetermined threshold considered to satisfy | fill the said reproduction | regeneration standard is more than a predetermined area as said imaging | photography failure candidate area. Video editing method.   In the candidate section detection step, a plurality of sections in which the interval at which a shooting failure frame or field in which the evaluation value satisfies the reproduction criterion is less than a predetermined threshold is less than the predetermined interval appears is a predetermined section. 5. The moving image editing method according to claim 4, wherein, when the number of times becomes above, the section is detected as the shooting failure candidate section.

Images