JP2005348328A - Video camera photographing support program and editing support program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To call attention in real time to user's camera work at a time of video camera photographing in a video camera photographing support program and an editing support program, and obtain a photographing image which includes many dynamic images suitable for editing. <P>SOLUTION: The camera work, in photographing a dynamic image, is analyzed in real time (S2) by projecting luminance of each pixel in each frame image which constitutes the photographed dynamic image in a horizontal direction and a vertical direction, and comparing the amount of luminance projection of the horizontal direction and the vertical direction between respective frame images. And, navigation for calling attention about the user's camera work, is performed (S3, S4) based on image grammar using the analysis result. By this means, it is possible to obtain photographing images which include many dynamic images suitable for user's editing. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a shooting support program for supporting shooting of a video camera by a user and an editing support program for supporting editing of a moving image shot by a video camera.

  In recent years, inexpensive digital video cameras have become widespread, and general users can easily shoot moving images. In addition, digitalization makes it possible to handle moving images in a non-linear manner, and various moving image editing software is on the market. However, it is rare for general users to perform editing work, and at present, a large amount of moving images are left unedited in each home. There are two possible causes for this.

  First, the first cause is lack of knowledge of moving image editing. In the moving image editing work, first, shots that have failed in shooting are excluded, and then one shot is selected from a plurality of shots in consideration of the relationship with previous and subsequent shots. Then, the shots are cut to an appropriate length without losing the amount of information and without redundancy, and the shots are connected. There are an infinite number of ways to select these shots and determine the length of the shots, but there are some universal rules when edited to accurately convey to the viewer what the producer intended. Exists. This is called video grammar. However, since general users do not have the opportunity to learn this video grammar and do not know which shot to use for editing and which shot should be used for how many seconds, moving image editing is difficult. It has become.

  The second cause is that there are few moving images suitable for editing in the captured moving images. Originally, a moving image should be taken in consideration of the above-mentioned video grammar, but it is very difficult for a general user to take an image while being aware of editing. For this reason, at the stage of editing the captured moving image, it is noticed that there is no shot suitable for editing in the captured moving image.

  By the way, in the field of instructional materials for moving image shooting techniques, the video shot by the user is shown on the display, and within this video, the best techniques such as best composition, zoom, pan, chill, etc. are used by professional photographers. , It is known that you can capture professional cameraman's shooting techniques easily and at low cost by shooting with a video camera and applying smoke to the part that the professional cameraman did not shoot ( For example, see Patent Document 1).

  However, in the method of displaying the video shot by the user as described above on the display and having the professional photographer recapture the video within this video, the intention of the producer side is truly in line with the above video grammar. It is difficult to take a shot that can accurately tell the viewer. Also, it is not possible to call attention to the user's camera work in real time when shooting with the video camera.

  In addition, in the field of video editing support systems, a video expression technique pattern is extracted from video data in a video database, and a video image newly shot by a user meets a specific expression intention based on the pattern. There is known one that provides a video expression technique (see, for example, Patent Document 2).

However, there is no guarantee that the video data stored in the video database is ideal in the video editing support system that extracts the pattern of the video expression technique that matches the specific expression intention from the video data in the video database. Moreover, since the video expression technique that can be provided by this video editing support system is limited to the number and length of scenes, the user cannot take a shot that is truly suitable for editing using this system. . Further, in the present invention, as in the invention disclosed in Patent Document 1, it is not possible to alert the user's camera work in real time when the video camera is photographed.
Japanese Patent Laid-Open No. 6-4016 (page 1-4, FIG. 1) JP 2002-262219 A (page 1-5, FIG. 1-5)

  The present invention has been made in order to solve the above-mentioned problem, and is capable of calling a user's camera work in real time when shooting with a video camera, and is suitable for editing. An object of the present invention is to provide a shooting support program for a video camera capable of obtaining a shot image including many images. In addition, the present invention can obtain a captured image including a large number of moving images suitable for editing so that the user's camera work can be alerted in real time when the video camera is captured. Moreover, it is an object of the present invention to provide an editing support program for a video camera that can instantly grasp which part of a captured moving image is suitable for editing.

  In order to achieve the above object, the invention of claim 1 provides a processor for supporting shooting by a video camera, the luminance of each pixel in each frame image constituting a moving image shot by the shooting means in the horizontal direction and Photographed by camera work analysis means and photographing means for analyzing in real time the camera work when moving images are shot by projecting in the vertical direction and comparing the luminance projection amounts in the horizontal and vertical directions between the frame images When the video is edited using the recorded video, the analysis by the camera work analysis means based on the video grammar, which is a universal rule required for the editor to accurately convey the intention to the viewer Judgment means for determining whether or not navigation for calling attention is necessary for the resulting camera work, and navigation by the judgment means When the emission is determined to be necessary, a video camera photographing support program to function as output means for outputting a signal for navigation.

  According to a second aspect of the present invention, in the video camera photographing support program according to the first aspect, the determination means includes at least a portion corresponding to severe camera shake in the camera work obtained as an analysis result by the camera work analysis means. When panning (shooting while rotating the camera), if the camera rotation speed is too fast, zooming is too fast, panning moving images (pan images) or zooming If a fixed image (moving image taken from a fixed viewpoint) of 1 second or longer is not taken before or after the moving image (zoom image) at the time of processing, it exists at a position other than before or after the pan image or zoom image. If the fixed image has not been shot for more than 3 seconds, and there are shots that are too bright or too dark, this camera Respect, it is to determine that it is necessary to navigation for attention.

  According to a third aspect of the present invention, there is provided the video camera photographing support program according to the second aspect, wherein the processor further captures the camera work based on the analysis result of the camera work analysis means with the start point and end point of the camera work as a boundary. Shot classification means for classifying moving images shot by the above, shots of fixed images that do not cause severe blurring of shots classified by the shot classification means, and videos during normal (not too fast) pan processing Unusable due to image shots, moving shots during normal (not too fast) zoom processing, moving image shots during pan processing too fast, moving image shots during zoom processing too fast, and severe image blur Indexing means for labeling shots of fixed images, etc. Sometimes, based on the result of labeling each shot by the indexing means, out of fixed image shots that do not have intense image blurring, the time normal rate, which is the ratio of shots with sufficient shooting time, and out of pan image shots The pan normal rate, which is the ratio of normal pan image shots, and the zoom normal rate, which is the ratio of normal zoom image shots out of zoom image shots, and severe image blurring among fixed image shots. Normal rate calculation means for calculating the normal rate of blur, which is the ratio of shots of fixed images that are not present, shooting based on the normal rate time, pan normal rate, zoom normal rate, and normal blur rate calculated by the normal rate calculation unit A score calculation means for calculating a score for a moving image photographed by the means, and a normal time calculated by a normal rate calculation means , The pan normal rate, the zoom normal rate or the blur normal rate function as normal rate determination means for determining whether or not the predetermined threshold value or less, and the output means represents a signal representing a calculation result by the score calculation means; A signal representing guide information for calling attention to an item determined to be equal to or less than a predetermined threshold by the normal rate determination means is output.

  According to a fourth aspect of the present invention, in the video camera photographing support program according to the third aspect, the type of the event to be photographed by the video camera is determined by the user using the processor and the operation means in the video camera. Based on the reading means for reading the shot data previously taken by the professional photographer for the event corresponding to the selected content from the storage device in the processor and the data of the shot read by the reading means when selected. It functions as shot output means for outputting a signal for displaying a shot to display means in the video camera.

  According to the fifth aspect of the present invention, a processor is provided to support editing of a moving image photographed by a video camera, and the luminance of each pixel in each frame image constituting the moving image photographed by the photographing means is set in the horizontal direction and the vertical direction. By projecting in the direction and comparing the horizontal and vertical luminance projection amounts between the frame images, the camerawork analysis means for analyzing the camerawork in real time when the moving image is photographed is photographed by the photographing means. Result of camera work analysis based on video grammar, which is a universal rule required for editors to accurately convey their intentions to viewers As a result of the determination, it is necessary to determine whether navigation is required to call attention to the camerawork obtained as a result of the analysis. Based on the start point and end point of camera work, the shot classification means for classifying the moving images shot by the shooting means into shots, and the shots classified by the shot classification means are free from severe blurring of the video Shots of images (moving images taken from a fixed viewpoint), moving images during normal (not too fast) panning, moving images during normal (not too fast) zooming, too fast Indexing means for labeling shots of moving images during pan processing, shots of moving images during zoom processing that is too fast, and shots of fixed images that cannot be used due to severe video blurring, and the result of labeling each shot by indexing means Based on whether the fixed image shot was taken for 3 seconds or longer The shots before and after the moving image shot during panning are either fixed image shots with a shooting time of 1 second or longer, or the shots before and after the moving image shot during normal zoom processing are both Detected using recommended section detection means and recommended section detection means that detect sections of fixed image shots with a shooting time of 1 second or longer as recommended sections that should be used when editing moving images shot with a video camera. Outputs a signal for displaying the recommended section on the display means in the video camera, and outputs a navigation signal in real time when the determination means determines that navigation is necessary A video camera editing support program for determining whether the determination means is a camera work analysis means. If there is at least a part corresponding to severe camera shake in the camera work obtained as a result of the analysis, the camera rotation operation speed during pan processing is too fast, the zoom processing operation is too fast, pan processing Fixed image that exists at a position other than before or after the panned image or zoomed image when a fixed image of 1 second or longer is not photographed before or after the moving image (panned image) at the time or the moving image (zoomed image) at the time of zoom processing Is not photographed for 3 seconds or more, and when there are shots that are too bright or too dark, it is determined that navigation is required to call attention to this camera work.

  According to a sixth aspect of the present invention, in the video camera editing support program according to the fifth aspect of the invention, the processor is further configured to, based on the result of labeling each shot by the indexing unit, at the end of the moving image shooting by the shooting unit The ratio of normal time, which is the ratio of shots with sufficient shooting time among fixed image shots without intense video blur, and the ratio of normal pan image shots of moving image shots during pan processing Pan normal rate, zoom normal rate that is the proportion of normal zoom image shots among moving image shots during zoom processing, and proportion of fixed image shots that do not have intense video blur among fixed image shots The normal rate calculating means for calculating the blur normal rate, the time normal rate and the bread normal rate calculated by the normal rate calculating means A score calculating means for calculating a score for a moving image photographed by the photographing means based on the normality rate and the blur normal rate, and a time normal rate, a pan normal rate, a zoom normal rate calculated by the normal rate calculating means, or It functions as normal rate determination means for determining whether or not the blur normal rate is equal to or less than a predetermined threshold value, and the output means outputs a signal indicating the calculation result by the score calculation means and the normal rate determination means with a predetermined threshold value or less. And a signal representing guide information for calling attention to an item determined to be.

  According to a seventh aspect of the present invention, in the video camera editing support program according to the sixth aspect, the type of the event to be photographed by the video camera is further determined by the user using the processor and the operation means in the video camera. Based on the reading means for reading the shot data previously taken by the professional photographer for the event corresponding to the selected content from the storage device in the processor and the data of the shot read by the reading means when selected. It functions as shot output means for outputting a signal for displaying a shot to display means in the video camera.

  According to the first aspect of the present invention, it is possible to analyze the user's camera work in real time and perform navigation for alerting the user about the camera work. As a result, it is possible to obtain a captured image including many moving images suitable for editing by the user.

  According to the invention of claim 2, when the camera work obtained as an analysis result by the camera work analysis means includes at least a portion corresponding to severe camera shake, the speed of the camera rotation operation during pan processing is too high. If the zoom processing operation is too fast, if a fixed image of 1 second or longer is not photographed before or after the pan image or zoom image, a fixed image existing at a position other than the front or back of the pan image or zoom image is 3 seconds. When no shot is taken, and when there are shots that are too bright or too dark, navigation for calling attention to this camera work is performed. Thereby, the effect described in claim 1 can be obtained accurately.

  According to the third aspect of the present invention, a score based on the time normal rate, the pan normal rate, the zoom normal rate, and the blur normal rate for the captured moving image, and the normal rate is equal to or less than a predetermined threshold. It is possible to output guide information for calling attention to the normal rate. Thereby, while being able to objectively evaluate and score the camerawork in the moving image image | photographed by the user, the improvement point about this camerawork can be shown.

  According to the fourth aspect of the present invention, it is possible to display shot data previously captured by a professional photographer for an event corresponding to the content selected by the user on the display means in the video camera. Thereby, since it can image | photograph with the camera work and camera angle similar to a professional cameraman, the moving image which looks good can be obtained.

  According to invention of Claim 5, a user's camera work can be analyzed in real time and the navigation for calling the user the attention regarding the camera work can be performed. As a result, it is possible to obtain a captured image including many moving images suitable for editing by the user. In addition, whether the shot time of the fixed image shot is 3 seconds or more, or the shots before and after the moving image shot at the time of normal pan processing are both fixed image shots with a shooting time of 1 second or more, Recommended section that should be used when editing a video shot with a video camera, where the shots before and after the shot of the video during normal zoom processing are both fixed image shots with a shooting time of 1 second or longer Therefore, the user can instantly grasp which part of the captured moving image is suitable for editing based on an objective standard.

  According to the invention of claim 6, in addition to the effect of claim 5, the same effect as the effect of claim 3 can be obtained.

  According to the invention of claim 7, in addition to the effect of claim 6, the same effect as the effect of claim 4 can be obtained.

  The best mode for carrying out the present invention will be described with reference to the drawings. The present invention relates to a shooting support program for supporting shooting of a video camera by a user, and an editing support program for supporting editing of a moving image shot by a video camera, and a moving image suitable for editing is provided. A large number of captured images can be obtained, and moreover, it is possible to instantly grasp which part of a captured moving image is suitable for editing. In the present embodiment, an example in which the photographing support program and the editing support program according to the present invention are stored in the ROM of a digital video camera will be described. In addition, embodiment described below does not cover this invention, and this invention is not limited only to the following form.

  FIG. 1 shows a configuration of a digital video camera incorporating a photographing support program and an editing support program (hereinafter collectively referred to as a support program) according to the present embodiment. The digital video camera 1 includes a ROM 3 that stores a support program 2, a microprocessor 4 (processor) that reads various programs from the ROM 3 and controls the entire camera, and a CCD (Charge Coupled Device) that is an image sensor. 5 (imaging means), an A / D converter 6 for converting moving image data in the form of an analog signal output from the CCD 5 into a digital signal format, and a digital signal (R) output from the A / D converter 6 , G, B imaging signals) in the form of moving image data temporarily stored therein, and R, G, B imaging signal format moving image data read from the image memory 7 in the form of characters and other data And an NTSC encoder for converting R, G, B imaging signals via the OSD synthesis unit 8 into NTSC composite signals. 9 converts the output signal from the NTSC encoder 9 into an analog signal, and a D / A converter 10 to be outputted to an external output device such as a television. The digital video camera 1 also compresses the moving image data in the format of the R, G, and B image signals output from the OSD combining unit 8 and the MPEG compressed moving image read from the card memory 13. An MPEG compression / decompression unit 12 that performs data decompression processing, and a card memory 13 that stores moving image data MPEG-compressed by the MPEG compression / decompression unit 12 are provided. Further, the digital video camera 1 includes a liquid crystal finder 11 (display means) for displaying an image based on the R, G, and B imaging signals via the OSD synthesis unit 8, and an operation unit 14 including various operation keys. And have. The above-mentioned liquid crystal finder 11 is also used when displaying a moving image of shots previously taken by a professional photographer for various events stored in the ROM. The camerawork analysis means, determination means, output means, shot classification means, indexing means, normal rate calculation means, score calculation means, normal rate determination means, reading means, shot output means, and recommended use section detection means in the claims are: This is realized by the microprocessor 4 and the support program 2 described above.

  Next, processing of the microprocessor 4 based on the support program 2 will be described with reference to the flowchart of FIG. When shooting (S1) is started by the user, the moving image data in the analog signal format output from the CCD 5 is converted into a digital signal format by the A / D converter 6, and the digital signals (R, G) are converted. , B imaging signals) is temporarily stored in the image memory 7. The microprocessor 4 reads moving image data in the form of R, G, B imaging signals stored in the image memory 7 on the basis of the instruction of the support program 2, and in each frame image constituting the moving image. By projecting the luminance of each pixel in the horizontal direction and the vertical direction and comparing the luminance projection amounts in the horizontal direction and the vertical direction between the respective frame images, the camera work at the time of shooting the moving image is analyzed in real time ( S2).

  Next, the camera work analysis process shown in S2 will be briefly described with reference to FIG. This camera work analysis process uses a projection method, and as described above, the luminance of each pixel in each frame image constituting the captured moving image is projected in the horizontal direction and the vertical direction, and each frame is projected. By comparing the luminance projection amounts in the horizontal direction and the vertical direction between the images, the camera work when the moving image is photographed is analyzed.

  In general, a digital image is represented by a set of squares called pixels, and the vertical and horizontal lengths of the image are expressed as hundreds of pixels. In a digital image, colors are expressed by combining light of three colors R, G, and B, and each pixel is divided into R, G, and B in 256 steps from 0 to 255. keeping.

In addition, the luminance is a scale representing the brightness per unit area, and has a value equal to the brightness of a black and white image. Luminance is based on R, G, B imaging signals that make up a color image.
Luminance = Red x 0.30 + Green x 0.59 + Blue x 0.11
It can be calculated by the following formula.

The luminance of each pixel is obtained using the above formula, and the average of the luminance of one column or one row is obtained for all rows and columns, thereby dropping the characteristics of one image into two graphs (one From the three-dimensional information of vertical x horizontal x luminance, two two-dimensional information of vertical x luminance and horizontal x luminance is obtained). This is the luminance projection amount, and the projection amount is high for rows and columns where many whitish pixels are present, and the projection amount is low for rows and columns where many blackish pixels are present. For example, in the case of the frame 21 shown in FIG. 3, the presence of the person 23 on the line 22 and the presence of the person 23 and the car 25 on the line 24 affect the projection amount. 24, a difference in projection amount due to the presence or absence of the vehicle 25 appears. In actual work, since this work is performed for all the rows of the image, the curve changes as shown by the graph 26 in FIG. As described above, there are variations in bright rows and columns, dark rows and columns in each frame image constituting the captured moving image, and the characteristics are shown in the horizontal × luminance graph 26 or the vertical × luminance graph. (See graph P _Y (f, i) in FIG. 5). By extracting and comparing this feature amount for each frame image, the user's camera work can be analyzed.

  In the camera work analysis processing described above, calculation of the movement amount by pan processing in the vertical and horizontal directions (processing for shooting while rotating the digital video camera 1), calculation of the zoom amount in the zoom processing, and calculation of the brightness of each frame are performed. Done.

  Of the above-described processes, the movement amount is calculated by panning in the horizontal direction as follows. First, as shown in FIG. 4, a vertical x luminance graph 31 representing the projection amount in the frame image 21 before panning processing and a vertical x luminance graph 33 representing the projection amount in the frame image 32 after panning processing. Are searched and a position having the most similar feature is searched for while shifting left and right within a range of ± 10% of the image width. In order to superimpose the graph 31 before pan processing and the graph 33 after pan processing, the central portion 34 of the graph 31 before pan processing (from the respective ends by the maximum amount of movement that can be taken from the frame image). The area 34 ′ having the same size as the central portion 34 in the graph 33 after the pan processing and the central portion 34) after panning is used. Then, a position with the most similar feature is searched while shifting left and right within a range of ± 10% of the image width. Actually, the luminance projection amount difference is calculated for each vertical column while being shifted to the left and right by one pixel within the range of ± 10% of the image width and summed up, and the place where the total value becomes the minimum is obtained by pan processing. Estimated travel. In the case of FIG. 4, at the position A shifted to the left, the difference in projection amount between the central portion 34 of the graph 31 before panning and the area 34 ′ in the graph 33 after panning is minimized, so It is estimated that panning (moved to the right by panning) is performed. Actually, not only the positions A, B, and C shown in the figure, but also the detection of the area 34 ′ where the difference in projection amount is the minimum at approximately 60 to 120 positions within the range of ± 10% of the image width. Processing is performed. The amount of movement due to the panning process in the vertical direction is estimated by the same operation. In addition, the zoom amount estimation processing in the zoom processing enlarges or reduces the length of the luminance projection amount, and, similarly to the calculation of the movement amount by the pan processing described above, the sum of the luminance differences is the minimum value. Is estimated as the zoom amount. Camera work analysis methods using these projection methods are excellent in processing speed. Therefore, it is possible to analyze the camera work in real time during shooting using the camera work analysis method using this projection method.

Next, details of the camera work analysis process using the above-described projection method will be described using mathematical expressions. First, the luminance projection amount will be described using mathematical expressions. FIG. 5 shows the concept of creating a projection distribution. In the figure, for the image of the frame f, the distribution projected in the horizontal direction is represented by P _X (f, j), and the distribution projected in the vertical direction is represented by P _Y (f, i). i takes a range from 1 to the width w of the frame image, and j similarly takes a range from 1 to the height h of the image. Frame f and P _X (f, j) in the figure correspond to the frame 21 in FIG. 3 and the projection distribution in the graph 26, respectively. When the luminance value of the coordinates (i, j) in the image frame f is expressed by Gray (f, i, j), each value of the luminance projection amounts P _X (f, j), P _Y (f, i) is These can be expressed as the following formulas (1) and (2).

Next, the movement amount estimation process (pan determination process) by the pan process will be described using mathematical expressions. In this pan determination process, the amount of movement in the vertical and horizontal directions between two frames is calculated. FIG. 6 shows an outline of movement amount calculation. The frame f, the frame (f + k), P _Y (f, i), and P _Y (f + k, i) in the figure are the projection distribution in the frame 21, the frame 32, and the graph 31 in FIG. Corresponds to the projection distribution. By comparing the projection distributions P _Y (f, i) and P _Y (f + k, i) obtained from the temporally adjacent frames f and (f + k), respectively, the frame f to the frame (f + k) ) To calculate the horizontal movement amount Pan _lr . Further, the vertical movement amount Pan _ud is calculated by comparing P _X (f, j) and P _X (f + k, j). For the comparison in the horizontal direction, only the central part excluding from the both ends by the maximum amount of movement d _{lr that} can be taken from the frame f is used, and the projection distribution of the frame (f + k) is shifted in the horizontal direction within the range of ± d _lr pixels. Make comparisons. Similarly, for the comparison in the vertical direction, only the central portion excluding from the top and bottom by the maximum amount of movement d _{ud that} can be taken from the frame f is used, and the projection distribution of the frame (f + k) is vertical within the range of ± d _ud pixels. Make comparisons while shifting in the direction. The amount of shift when the correlation is the highest is the amount of movement in the vertical and horizontal directions, Pan _ud and Pan _lr . Here, the vertical and horizontal projection distances between the two frames are calculated as the following equations (3) and (4), respectively. The horizontal movement amount Pan _lr and the vertical movement amount Pan _ud are respectively It can define as Formula (5) and (6). Note that _{δ lr, (δ lr = -d} lr, ··, 0, ·· d lr) and the [delta] _{ud is, (δ ud = -d ud,} ··, 0, ·· d ud) Are used respectively. If the minimum distance is more appeared, corresponding [delta] _lr, among [delta] _ud, prior frame same sign of [delta] _lr, the [delta] _ud as priority, also the smallest [delta] _lr absolute value, to select the [delta] _ud. This is equivalent to prioritizing small movements.

Next, zoom amount estimation processing (zoom determination processing) in the above zoom processing will be described using mathematical expressions. FIG. 7 shows an overview of zoom amount calculation. A frame f in the figure corresponds to the frame 21 in FIG. First, the horizontal projection distribution P _X frame f determined in Equation (2), assuming the amount of zoom that can be taken, larger partial only that assumption zoom amount, creating a reduced projection distribution diagram. The enlarged / reduced projection amount is shown in the following formula (7).

If the maximum zoom amount that can be taken is d _z , δ _z in equation (7) above changes in units of one pixel within the range of (−d _z ,..., 0,... D _z ) pixels. Therefore, 2 × d _z +1 projection distributions P _z are created. The value of d _z is determined by the maximum zoom speed to be accommodated.

The zoom amount is calculated by comparing each of the thus obtained projection distributions with the projection distribution P _X (f + k, j) obtained from the adjacent frame images. The comparison, only the maximum amount of zoom d _z component that can take use of only a central portion except each from above and below, obtains the zoom amount when creating the most dissimilarity lower projection distribution. The following formula (8) is a formula for obtaining this zoom amount.

  Similar to the above, the zoom ratio in the vertical direction can be obtained, but the number of vertical pixels is smaller than the horizontal. For this reason, it is expected that the feature of the enlargement / reduction change due to the zoom is difficult to appear in the projection amount, and the detection accuracy may be deteriorated. In addition, considering that the zoom amount is sufficiently calculated only from the horizontal direction, the extra calculation slows down the processing speed, and in this embodiment, the zoom ratio in the vertical direction is not obtained.

The zoom magnification can be calculated by the following equation (9).

However, since enlargement is calculated as a real number greater than 1, and reduction is calculated as a real number greater than 0 and less than 1, it cannot be compared with the pan amount having a positive or negative value. Therefore, in order to handle it in the same row as the pan amount, conversion is performed using the following equation (10). In equation (10), h _d = 600 is used to match the difference in scale with the amount of bread.

Next, the brightness calculation for each frame will be described using mathematical expressions. The brightness of each frame can be expressed as the sum of luminance projection amounts P _Y (f, i) shown in the above equation (2) from i = 1 to i = w. The brightness of the frame f is shown in the following formula (11).

  Next, the description returns to the processing in the flowchart of FIG. Whether the microprocessor 4 needs navigation for alerting the user's camera work based on the analysis result obtained in the camera work analysis process shown in S2 based on the instruction of the support program 2 Is determined (S3). If it is determined that the navigation is necessary, for example, if the camera work obtained as a result of the camera work analysis process includes a portion corresponding to intense camera shake, pan processing (process that performs shooting while rotating the camera) ) If the camera rotation speed is too fast, or if the zoom process is too fast, the moving image (pan image) during pan processing or the moving image (zoom image) during zoom processing is 1 second or longer. Fixed images (moving images taken from a fixed viewpoint) are not taken, fixed images existing at positions other than before and after the pan image or zoom image have not been taken for more than 3 seconds, and too bright This is the case when there are shots or shots that are too dark. If the microprocessor 4 determines that navigation is necessary (YES in S3), depending on the reason for determining that navigation is necessary, “the blurring of the video is extremely severe”, “the camera work is too fast” , Navigation information such as “a fixed image was not taken for a sufficient time” is displayed on the liquid crystal finder 11 (S4).

  The microprocessor 4 repeats the processes of S1 to S4 described above until shooting by the user is completed (NO in S5). Then, based on the result of the camera work analysis shown in S2, the moving images shot in the process of S1 are classified into shots with the start point and end point of the camera work as boundaries. Then, for each shot after classification, a shot of a fixed image that does not cause intense image blur, a shot of a moving image during normal (not too fast) pan processing, and a video during normal (not too fast) zoom processing Labeling (indexing) includes shots of images, shots of moving images during pan processing that is too fast, shots of moving images during zoom processing that is too fast, and shots of fixed images that cannot be used due to severe blurring of images (S6).

  When the labeling for each shot is completed, the microprocessor 4 is a section that is preferably used when editing the moving image captured in S1, based on the data (index information) of the labeling result for each shot. A certain recommended section is detected. Specifically, the microprocessor 4 reads the above index information, and there is a normal shot section (hereinafter referred to as a “Fix section”) of the fixed image in the index information. Whether or not to search is searched in order from the top of the index information. Then, when a Fix section is detected (YES in S7), the next shot section of the detected Fix section is a normal (no intense video blurring) shot section of the fixed image (Fix section) and normal ( It is determined whether or not it is a moving image shot section (Pan section) during pan processing or a normal (not too fast) moving image shot section (Zoom section) during pan processing. (S8). If the result is normal (YES in S8), the microprocessor 4 further determines whether or not the next shot section is a Fix section (S9). If the result of the determination in S9 is a Fix section (YES in S9), the Fix section detected in S7 and the section targeted for determination in S8 and S9 are detected as recommended sections ( S10). Specifically, when three consecutive shot sections become a Fix section-Pan section-Fix section, Fix section-Zoom section-Fix section, or Fix section-Fix section-Fix section, The three shot sections to be detected are detected as recommended sections.

  The reason for adopting the recommended section determination method as described above is as follows. In other words, fixed image shots are easy to see and understand without leaving ambiguity, and shots during pan processing and zoom processing leave a lot of ambiguity, so when performing pan processing and zoom processing, This is because, in order to reduce ambiguity, it is desirable that there is a Fix section before and after the Pan section and the Zoom section.

  Note that, as described above, based on whether or not three consecutive shot sections are in any combination of the above, a recommended section is not detected, but the time interval of each shot section is determined. It is desirable to detect the recommended section by paying attention to the length. For example, when the combination of three consecutive sections is a Fix section-Fix section-Fix section, it is desirable to set this section as a recommended section when the total shooting time of these sections is 3 seconds or more. . In addition, when the combination of three consecutive sections is Fix section-Pan section-Fix section, or Fix section-Zoom section-Fix section, the shooting time of the Fix section before and after the Pan section or Zoom section is all It is desirable that it is 1 second or more.

  Based on the instruction of the support program 2, the microprocessor 4 outputs and displays on the liquid crystal finder 11 frame images and shooting times corresponding to the start and end points of the recommended section detected in the processes of S7 to S10. S11). In the present embodiment, it is also possible to reproduce these recommended section shots on the liquid crystal finder 11 and visually display the result of the recommended section detection processing on the liquid crystal finder 11 as a graph.

  The microprocessor 4 performs the following scoring process together with the recommended section detection process. That is, the microprocessor 4 reads the index information obtained as a result of the indexing process of S6 based on the instruction of the support program 2, and the temporal sufficiency (normality) of the Fix section in the index information Is determined (S12). However, since the length of time required for the fixed image varies depending on the sense of distance of the video, the temporal sufficiency of the Fix section is determined in consideration of the sense of distance of each shot.

  The above point will be described in detail. In video grammar, the sense of distance of video is roughly divided into three types: loose shot (LS), medium shot (MS), and tight shot (TS). These are relative shot concepts. A shot closer to the target than a certain shot is called a tight shot, a shot farther from the target than a certain shot is called a loose shot, and a middle shot is called a medium shot. In general, an image taken from a greater distance has more information in one screen than an image taken from a distance. Therefore, more time is required for a person to recognize this. The time required for recognition is determined empirically. In the video grammar, based on this classification, the length of time required for a fixed image is 6 seconds for LS, 4 seconds for MS, and TS for TS. It is defined as 2.5 seconds. Since these are relative definitions, in this embodiment, shot classification is performed by checking whether an image exists as a part of another image for each Fix section. Yes. Then, the temporal sufficiency of the Fix section is determined according to the classification result.

  Next, based on the instruction of the support program 2, the microprocessor 4 statistically calculates the index information obtained in the process of S6, based on the determination result of the temporal normality of the Fix section shown in S12. (S13). Then, based on the result of the statistical process, a score relating to the moving image obtained by the photographing process of S1 is calculated (S14).

Next, the score calculation process will be specifically described. For example, assume that the result of the statistical processing of the index information is as follows.
Fix: 6 The total number of Fix sections in the index information is normal (no intense video blur).
Pan: 9 The total number of normal Pan sections is 9.
Zoom: 3 The total number of normal Zoom sections is 3.
Can't use: 4 There are 4 sections that are judged to be unsuitable for use due to severe blurring of the video
Too fast pan: 3 Pan section determined to be too fast
Too fast zoom: 7 The zoom section determined to be too fast is 7
Bright: 0 The area that is too bright is 0
Dark: 0 The section that is too dark is 0

  The microprocessor 4 obtains an intermediate value from which the score is calculated based on the result of the statistical processing of the index information. First, a time normal rate that is a ratio of shots with sufficient shooting time among fixed image shots (ratio of sections with sufficient shooting time among Fix sections) is obtained. Specifically, assuming that there are four Fix sections determined to be normal as a result of the determination of temporal normality of the Fix section shown in S12, the number of Fix sections that are temporally normal Based on the above and the total number of Fix sections in the above index information (there is no intense video blur), the time normality ratio is 4 / 6≈66%.

  Also, the ratio of normal pan image shots among the shots of moving images (pan images) during pan processing (the ratio of normal Pan sections out of the total of Pan sections determined to be too fast and normal Pan sections) ) Pan normal rate is Pan / (Pan + Too fast pan) = 9 / (9 + 3) = 75%. Furthermore, the ratio of normal zoom image shots among the shots of the moving image (zoom image) during zoom processing (the ratio of the normal Zoom section to the normal Zoom section and the Zoom section determined to be too fast) The normal zoom ratio is Zoom / (Zoom + Too fast zoom) = 3 / (3 + 7) = 30%. In addition, of fixed image shots, the ratio of fixed image shots without intense video blur (normal (no intense video blur) Fix segment and segments determined to be unsuitable for use due to severe video blur The blur normal rate, which is the ratio of the normal Fix interval), is Fix / (Fix + Can't use) = 6 / (6 + 4) = 60%.

  The microprocessor 4 calculates a score for the moving image captured in S1 based on the time normal rate, the pan normal rate, the zoom normal rate, and the blur normal rate. Then, together with the above score, shooting guide information useful for improving the score is output to the liquid crystal finder 11 (S15). The liquid crystal finder 11 displays these scores and shooting guide information. The shooting guide information is determined, for example, when the time normal rate, the pan normal rate, the zoom normal rate, or the blur normal rate is determined to be equal to or lower than a predetermined threshold, or to be equal to or higher than a predetermined threshold. Is displayed on the liquid crystal finder 11. In the case of the above example, since the zoom normal rate is not so good, guide information for prompting the user to pay attention to zoom is displayed on the liquid crystal finder 11.

Next, how the user's shooting score is improved by the support program 2 will be described based on experimental results. In this experiment, 10 subjects took 2 to 3 shots each for about 2 minutes. The first time, I was asked to shoot arbitrarily without giving instructions regarding the shooting method. The second time was aimed at improving the score according to the first time score and guide. The third time was aimed at further improvement based on the results of the first and second times. In the following experiment, camera work detection was performed 6 times per second, that is, every 1/6 second. In this experiment, the following information was displayed as the photographing guide information shown in S15 of FIG.
(A) A fixed image is taken with a sufficient length (3 seconds or longer) (b) Many fixed images have a short shooting time (less than 3 seconds) (c) Pan operation at an appropriate speed (past 1 Many shots can be taken with a pan operation that does not move more than 15% of the width or height in a second) (d) A pan operation that is too fast (a pan that moves more than 15% of the width or height in the past 1 second) (E) Zooming at an appropriate speed (zooming without zooming at 1% or more in the past 1 second) (f) Zooming too fast (past 1) (Zoom operation with a zoom of 1% or more) in a second) (ki) Image blurring severely (camera work stability is low. That is, the maximum and minimum pan amount (movement amount) in the past 1 second) The difference in values That more than 15% of the width and height of beam image. Specific examples will be described later.) Often (h) compared with the past score (second and subsequent)

  Note that 15% of the width in (D) corresponds to 48 pixels in the case of a frame image having a width of 320 pixels. Further, the “maximum value and minimum value of the pan amount (movement amount) in the past one second” shown in (ki) above is, for example, the horizontal pan amount (movement amount) in the camera work in the past one second. If 1 pixel on the right, 2 pixels on the right, 1 pixel on the right, 6 pixels on the right, 6 pixels on the right, and 7 pixels on the right, 7 pixels and 1 pixel are detected. Therefore, the “difference between the maximum value and the minimum value of the pan amount (movement amount) in the past one second” shown in (g) is 6 pixels. In this example, if the horizontal width of the frame image is 320 pixels, the difference is within 15% of the horizontal width of the frame image.

  As shown in FIG. 8 and FIG. 9, in the second experiment, it was confirmed that 9 out of 10 scores were improved. In the third experiment, all four people improved their scores, confirming the effectiveness of the system.

  Next, the pattern guide installed in the digital video camera 1 storing the support program 2 will be described. In general, when an image is taken by an amateur, there are problems such as not knowing what to take, not knowing how to take, and omission of shooting. Conventionally, in order to solve these problems, there are only methods in which a user reads a textbook for shooting and learns from a good person. The support program 2 can display what kind of video should be taken on the liquid crystal finder 11 of the digital video camera 1 to solve the above-mentioned problem.

  Specifically, a snapshot of a video shot by a professional photographer is prepared in advance, and a series of scenes to be shot is displayed on the liquid crystal finder 11 in a menu. For example, in the case of wedding photography, thumbnail images of scenes such as “Bride and Groom entrance scene” shown in FIG. 10A and “Matcha greeting scene” shown in FIG. 10B are displayed in the menu. The When the user selects a thumbnail image of one of the scenes on the menu, a full-scale image of the selected thumbnail image is displayed on the liquid crystal finder 11 and the composition of shooting is shown. In accordance with this composition, the user shoots at a similar angle, returns to the menu, and selects a thumbnail image of the scene to be shot next. Thereby, even if it is not a professional cameraman, it can image | photograph with the camera work and camera angle similar to a professional cameraman, and can image | photograph the image which looks good. Such guidance snapshot data is prepared for each typical event such as “birthday”, “athletic meet”, “wedding”, and stored in the ROM 3.

  On the other hand, at the time of daily shooting, not an event, the pattern guide program included in the support program 2 presents general shooting principles to the user. For example, at the start of shooting, an instruction is given to take a long shot for 6 seconds in order to show “where (far view of the scene)”. Next, in order to show “who”, it is instructed to take a middle shot for 4 seconds. Further, in order to show “what”, the user is instructed to take a tight shot for 3 seconds. After that, it instructs to shoot with a combination of middle shot and tight shot. In this case, even if it is said to be a middle shot or a long shot, it is not known to an amateur, so an image 81 containing marks 82 to 84 and time 85 as shown in FIG. 11 is displayed on the liquid crystal finder 11. The marks 82, 83, and 84 on the image 81 correspond to a long shot, a middle shot, and a tight shot, respectively. Therefore, the marks 82 to 84 and the time 83 on the image 81 mean that a long shot is first taken for 6 seconds, then a middle shot is taken for 4 seconds, and then a tight shot is taken for 3 seconds.

  As described above, according to the digital video camera 1 incorporating the support program 2 according to the present embodiment, the user's camera work is analyzed in real time, and navigation is performed to alert the user about the camera work. Can do. As a result, it is possible to obtain a captured image including many moving images suitable for editing by the user. In addition, for a captured moving image, it is possible to output a score based on a normal time rate, a normal pan rate, a normal zoom rate, and a normal shake rate, and guide information for improving the score. Thereby, the camera work in the moving image photographed by the user can be objectively evaluated and scored, and the improvement points regarding the camera work can be presented. Furthermore, it is possible to display on the liquid crystal finder 11 shot data previously photographed by a professional photographer for an event corresponding to the content selected by the user. Thereby, since it can image | photograph with the camera work and camera angle similar to a professional cameraman, the moving image which looks good can be obtained. In addition, since it is possible to detect a recommended section that should be used when editing a moving image, the user can instantly determine which part of the captured moving image is suitable for editing based on objective criteria. I can grasp it.

  In addition, this invention is not restricted to the said embodiment, Various deformation | transformation are possible. For example, in the above embodiment, the support program 2 is stored in the ROM 3 in the digital video camera 1 and executed by the microprocessor 4 in the digital video camera 1. However, the digital video camera and a personal computer are connected. The support program may be operated on the personal computer side by connecting. In the above embodiment, when the score is output and displayed on the liquid crystal finder 11, only the shooting guide information is displayed together with the score. However, the time normal rate, the pan normal rate, the zoom normal rate, and a part of the blur normal rate are included. Or you may make it display all on the liquid crystal finder 11 with a score. Further, the number of detected recommended sections may be displayed together with these pieces of information. In the above-described embodiment, navigation, a score, and shooting guide information for calling attention to the user's camera work are displayed on the liquid crystal finder 11, but these information are displayed on the liquid crystal monitor of the digital video camera. You may make it output, and you may make it output with an audio | voice.

The block diagram of the digital video camera which incorporated the imaging | photography assistance program and edit assistance program by this embodiment. The flowchart of the process of the microprocessor based on the said assistance program. Explanatory drawing of the outline | summary of the camera work analysis process in FIG. Explanatory drawing of the outline | summary of the calculation process of the movement amount by the pan process included in the said camera work analysis process. The figure which shows the concept of the projection distribution preparation in the said camera work analysis process. The figure which shows the outline | summary of the movement amount calculation in the said camera work analysis process. The figure which shows the outline | summary of the zoom amount calculation in the said camera work analysis process. The table | surface which shows how the score at the time of a user's imaging | photography improved by the said assistance program. The graph which shows how the score at the time of a user's imaging | photography improved by the said assistance program. (A) and (b) are the figure which shows the image of the scene of "the entrance scene of the bride and groom", respectively, and the figure which shows the image of the scene of the "greeting scene of the matchmaker". The figure which shows the example of the image which the program for pattern guides contained in the said assistance program displays on a liquid crystal finder at the time of daily imaging | photography.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Digital video camera 2 Support program (Shooting support program, edit support program, camera work analysis means, determination means, output means, shot classification means, indexing means, normal rate calculation means, score calculation means, normal rate determination means, reading means , Shot output means, recommended use section detection means)
4 Microprocessor (processor, camera work analysis means, determination means, output means, shot classification means, indexing means, normal rate calculation means, score calculation means, normal rate determination means, reading means, shot output means, recommended use section detection means )
5 CCD (photographing means)
11 Liquid crystal finder (display means)

Claims (7)

A processor to assist with video camera shooting,
By projecting the luminance of each pixel in each frame image constituting the moving image photographed by the photographing means in the horizontal direction and the vertical direction, and comparing the luminance projection amounts in the horizontal direction and the vertical direction between the respective frame images. , Camera work analysis means for analyzing camera work in real time when shooting the moving image,
Based on video grammar, which is a universal rule required for an editor to accurately convey his intention to a viewer when editing a video using a moving image shot by the shooting means, When it is determined by the determination means that navigation is necessary for the camera work obtained as an analysis result by the camera work analysis means, and when the determination means determines that navigation is necessary Output means for outputting a navigation signal,
Video camera shooting support program to function as.   When the camera work obtained as a result of analysis by the camera work analysis means has at least a portion corresponding to severe camera shake, the determination means is a camera at the time of pan processing (processing for taking a picture while rotating the camera). If the rotation operation speed is too fast, or if the zoom processing operation is too fast, a fixed image (one or more seconds) before or after the moving image (pan image) during pan processing or the moving image (zoom image) during zoom processing ( A moving image taken from a fixed viewpoint), a fixed image existing at a position other than before or after the pan image or zoom image has not been photographed for more than 3 seconds, and a shot that is too bright or too dark. When a shot exists, this camera work is determined to require navigation to call attention Videography support program according to claim 1, wherein the door. The processor further,
Based on the analysis result by the camera work analysis means, a shot classification means for classifying the moving images photographed by the photographing means into shots with the start point and end point of the camera work as a boundary,
For shots classified by the shot classifying means, fixed image shots without intense video blur, moving image shots during normal (not too fast) pan processing, and normal (not too fast) zoom processing Indexing means for labeling, such as shots of moving images, shots of moving images during pan processing too fast, shots of moving images during zoom processing that is too fast, and shots of fixed images that cannot be used due to severe image blur,
Based on the labeling result of each shot by the indexing means at the end of moving image shooting by the shooting means, the ratio of the shots with sufficient shooting time among the shots of the fixed image without intense video blurring Fix normal time, pan normal rate that is the proportion of normal pan image shots among pan image shots, zoom normal rate that is the proportion of normal zoom image shots among zoom image shots, and fix A normal rate calculating means for calculating a normal rate of blur, which is a ratio of a shot of a fixed image without intense video blur among shots of an image,
Score calculating means for calculating a score for a moving image photographed by the photographing means based on a normal time ratio, a pan normal ratio, a zoom normal ratio, and a blur normal ratio calculated by the normal ratio calculating means; and the normal The time normal rate, pan normal rate, zoom normal rate or blur normal rate calculated by the rate calculating means function as normal rate determining means for determining whether or not a predetermined threshold value or less,
The output means outputs a signal representing a calculation result by the score calculating means and a signal representing guide information for calling attention to an item determined to be not more than a predetermined threshold by the normal rate determining means. The video camera shooting support program according to claim 2, wherein: The processor further,
When the user selects a type of event to be photographed by the video camera using the operation means in the video camera, shot data previously photographed by a professional photographer for the event corresponding to the selected content As a shot output means for outputting a signal for displaying the shot to the display means in the video camera based on the shot data read by the reading means. The video camera shooting support program according to claim 3, wherein the video camera shooting support program is executed. A processor to assist in the editing of the video captured by the video camera,
By projecting the luminance of each pixel in each frame image constituting the moving image photographed by the photographing means in the horizontal direction and the vertical direction, and comparing the luminance projection amounts in the horizontal direction and the vertical direction between the respective frame images. , Camera work analysis means for analyzing camera work in real time when shooting the moving image,
Based on video grammar, which is a universal rule required for an editor to accurately convey his intention to a viewer when editing a video using a moving image shot by the shooting means, A determination means for determining whether or not navigation for calling attention is necessary for the camera work obtained as an analysis result by the camera work analysis means,
Based on the analysis result by the camera work analysis means, a shot classification means for classifying the moving images photographed by the photographing means into shots with the start point and end point of the camera work as a boundary,
For shots classified by the shot classification means, shots of fixed images (moving images taken from a fixed viewpoint) that do not have intense image blur, and moving images during normal (not too fast) pan processing Shots, shots of moving images during normal (not too fast) zoom processing, shots of moving images during pan processing that is too fast, shots of moving images during zoom processing that is too fast, and fixed images that cannot be used due to severe image blur Indexing means for labeling shots,
Based on the result of labeling each shot by the indexing means, either the section where the shooting time of the shot of the fixed image is 3 seconds or more, or the shot before and after the shot of the moving image during the normal pan processing, Is a section of a fixed image shot with a shooting time of 1 second or more, or a section where the shots before and after the shot of the moving image at the time of normal zoom processing are both shots of a fixed image with a shooting time of 1 second or more, Use recommended section detecting means for detecting a recommended section preferably used when editing a moving image shot by the video camera, and a recommended section detected by the recommended use section detecting means as display means in the video camera. A signal for display is output, and the determination means determines that navigation is necessary. A video camera editing support program for functioning as an output means for outputting a navigation signal in real time,
If the camera work obtained as a result of analysis by the camera work analysis means has at least a portion corresponding to severe camera shake, if the speed of the camera rotation operation during pan processing is too fast, If the processing operation is too fast, if a fixed image of one second or longer is not taken before or after the moving image (pan image) during pan processing or the moving image (zoom image) during zoom processing, the pan image or zoom image Navigation is required to call attention to this camera work when fixed images that exist in positions other than before and after have not been shot for more than 3 seconds, and when there are shots that are too bright or too dark An editing support program for a video camera, characterized in that it is determined that The processor further,
At the end of moving image shooting by the shooting means, based on the result of labeling each shot by the indexing means, the ratio of shots with a sufficient shooting time among the shots of the fixed image with no intense video blurring Normal time ratio, pan normal ratio that is the ratio of normal pan image shots out of moving image shots during pan processing, and ratio of normal zoom image shots out of moving image shots during zoom processing A normal rate calculation means for calculating a zoom normal rate and a blur normal rate, which is a ratio of a fixed image shot without a sharp video blur among fixed image shots,
Score calculating means for calculating a score for a moving image photographed by the photographing means based on a normal time ratio, a pan normal ratio, a zoom normal ratio, and a blur normal ratio calculated by the normal ratio calculating means; and the normal The time normal rate, pan normal rate, zoom normal rate or blur normal rate calculated by the rate calculating means function as normal rate determining means for determining whether or not a predetermined threshold value or less,
The output means outputs a signal representing a calculation result by the score calculating means and a signal representing guide information for calling attention to an item determined to be not more than a predetermined threshold by the normal rate determining means. 6. The editing support program for a video camera according to claim 5, wherein the program is made to function as shot output means. The processor further,
When the user selects a type of event to be photographed by the video camera using the operation means in the video camera, shot data previously photographed by a professional photographer for the event corresponding to the selected content As a shot output means for outputting a signal for displaying the shot to the display means in the video camera based on the shot data read by the reading means. 7. The video camera editing support program according to claim 6, wherein the video camera editing support program is executed.

Images