JP3327520B2 - Shooting method with NG warning function, shooting apparatus with NG warning function, and recording medium storing shooting program - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a photographing method with an NG warning function, a photographing device with an NG warning function, and a recording medium on which a photographing program is recorded. Regarding effective technology.

[0002]

2. Description of the Related Art With the advent of the multimedia era, video information can be recorded on media in digital form,
By combining with a non-linear editing machine that operates on a personal computer that can be used at home, it is possible for ordinary users to edit high-quality video. Under such an editing environment, as a method of obtaining a video material used for video editing, it is common to shoot with a video camera.

[0003]

However, the photographing operation using a video camera is easily affected by the surrounding environmental conditions, and requires a photographing skill of the photographer. For example, when the amount of light is not obtained due to nighttime and the image becomes black, the amount of light suddenly increases due to strong light, and when the image becomes pure white, the obstacle such as a pedestrian is not noticed. If you start shooting at the same time and those obstacles enter between the shooting subject and the video camera during shooting, unexpected accidents such as the photographer falling while moving while shooting while moving. If it occurs, or
Various obstacles are conceivable, such as a case where a screen shake appears as the video camera moves. In particular, with the remarkable miniaturization of consumer video cameras and the ability to shoot video with one hand, there are many problems that images are moved little by little due to camera shake. One of the problems when shooting the target object (video material), which is an obstacle during editing, is that when shooting with a video camera,
In most cases, the photographer is not conscious.

In order to solve the above-mentioned problems, many video cameras currently on the market are equipped with a camera shake preventing function for minimizing the effects of camera shake and reproducing the content of a shot image to reproduce the contents of the video image. There are two standard functions for playback on the. In this way, the video cameras currently on sale are equipped with a camera shake prevention function,
Image distortion due to camera shake has been removed to some extent,
Further, it is possible to judge the result of the photographing by the reproduction display function immediately after the photographing.

At present, consumer video cameras for writing digital video data to a hard disk are also manufactured and sold. In such a consumer video camera, since the recording medium is a hard disk, random access is possible, and it is easy to immediately reproduce the video immediately after shooting.

[0006] However, it is extremely rare for video cameras using tape media or disk media to reproduce and display video images shot by the video camera immediately after shooting, similarly. In this way, the reasons for not confirming the video shot with the video camera by playing it back on the video camera immediately after shooting are that it is troublesome to perform the playback operation, and that the video shot immediately after shooting is played back and displayed. In such a case, the photographing time and the reproduction time are required, which takes a long time, and the photographer does not think that there is such a problem at the time of photographing. That is, the photographer does not recognize that there is a problem when shooting with the video camera, but notices that there is a problem at the time of editing. However, even if you notice when editing,
Opportunities to shoot video material are often time- and space-constrained, often cannot be taken many times, and in many cases there is only one chance to shoot, so it is almost always difficult to shoot again Therefore, the editing of the video is advanced with the problematic material without solving the problem.

As one of the solutions for solving the above-mentioned problems, it is conceivable to add an editing function to the video camera so that the photographing operation and the editing operation can be performed simultaneously. In fact, video cameras with editing functions have been proposed, but editing is a difficult task even with the current non-linear editing machine, and it takes time to shoot while editing. Therefore, a video camera with an editing function is not a practical solution to the above problem.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems of the prior art, and an object of the present invention is to use it for editing work in a photographing method with an NG warning function and a photographing apparatus with an NG warning function. It is possible to output information that can determine whether the video is suitable for editing work during or immediately after shooting the target object so that the video that is the material can be shot while strongly considering the premise of shooting the video. It is to provide a technology that becomes possible.

Another object of the present invention is to provide a recording medium storing a photographing program for causing a computer to execute the photographing method with an NG warning function.

The above and other objects and novel features of the present invention will become apparent from the description of the present specification and the accompanying drawings.

[0011]

SUMMARY OF THE INVENTION Among the inventions disclosed in the present application, the outline of a representative one will be briefly described.
It is as follows.

Namely, the present invention provides an NG warning function imaging device for generating an easy video information available for editing, an imaging unit for photographing the object, temporarily the image captured by the imaging unit A continuous image memory unit for holding, an information extracting unit for extracting various information including brightness, color, camera work, movement of a subject, and image blur in the image held in the continuous image memory unit, and the information extraction A state determining unit that determines whether the image held in the continuous image memory unit is suitable for editing from the various information extracted by the unit; and the state determining unit that stores the image held in the continuous image memory unit. An NG state signal output unit for notifying that the image is not suitable for editing when it is determined that the image is not suitable for editing; and a data storage unit for storing the image held in the continuous image memory unit. An image display unit that displays at least one of an image at the time of shooting, an image held in a continuous image memory unit, or an image stored in a data storage unit, and a control unit that controls each unit. And

[0013] Also, various kinds of information extracted by the information extraction unit are stored in the continuous image memory unit together with the image captured by the imaging unit, and are also stored in the data storage unit.

Further, the present invention is a photographing method with an NG warning function for photographing an object and generating video information which can be easily used for editing.
Temporarily hold, brightness, color,
Camera work, movement of a subject, and various information including image blur are extracted, and from the extracted various information, it is determined whether the held image is suitable for editing, and the held image is edited. If it is determined that it is not suitable for
Notifying that the image is not suitable for editing, storing the stored image, and displaying at least one of image information at the time of shooting, the stored image, or the stored image. .

Further, the extracted various information is held and stored together with the photographed image.

[0016]

Embodiments of the present invention will be described below in detail with reference to the drawings.

In all the drawings for describing the embodiments, those having the same functions are denoted by the same reference numerals, and their repeated description will be omitted.

FIG. 1 is a block diagram showing a schematic configuration of a photographing apparatus with an NG warning function according to an embodiment of the present invention. As shown in the figure, in the photographing apparatus of the present embodiment,
The target object is photographed by the imaging unit 1. The continuous image captured by the imaging unit 1 is temporarily held by the continuous image memory unit 2. Note that a signal from an object input to the imaging unit 1 is an optical signal in a visible light region, an optical signal in an invisible region obtained by a filter or the like, an optical signal such as a laser signal,
Alternatively, special signals such as ultrasonic waves other than optical signals may be used. As long as the imaging unit acquires normal optical information, a photographing support function such as an autofocus function may be mounted, or may not be mounted. Further, in the case where the optical system is not a general optical system, it is equivalent to a general system corresponding to the processing system of the optical system.

Next, the information extracting section 4 analyzes the continuous image held in the continuous image memory section 2 to extract various information. Various types of information output from the information extracting unit 4 are temporarily stored in the continuous image memory unit 2 together with the continuous images captured by the image capturing unit 1. Various information output from the information extraction unit 4 includes brightness (luminance),
Information such as color, movement of a subject, camera work, blur of an image, and the like.

Next, the state discriminator 6 uses the various information extracted by the information extractor 4 as a parameter to determine whether or not the continuous image held in the continuous image memory 2 is sufficient for editing. That is, it is determined whether the user is not suitable for editing (ineligible for use (NG)) or suitable for editing (eligible for use (not NG)). In this case,
As the determination method in the state determination unit 6, the determination is performed using one or a plurality of pieces of information obtained by the information extraction unit 4. Next, the NG state signal output unit 8 outputs an NG signal according to the various information determined by the state determination unit 6 and determines that the image is not suitable for editing (NG), and is suitable for editing (NG). NG), then N
A signal other than the G signal is output. NG status signal output unit 8
Is a signal other than the NG signal, the continuous image is stored as it is in the data storage unit 3 such as a hard disk or a tape from the temporary holding state. In this case, various information is also stored in the data storage unit 3.

The image display unit 5 displays all of the photographed images, the continuous images in the continuous image memory unit, and the images stored in the data storage unit 3. In this case, the image display units 5 may each have a dedicated display device, or may be switched by a single device by some method and played back simultaneously. Further, the display target is not all information, and may be a reduced image of each image, a thinned time information, or a slightly processed one including or not including audio information.

When the signal from the NG state signal output section 8 is an NG signal, the NG signal is output to the photographer, for example, by displaying on the image display section 5 that the NG signal has been output. Announcing the information that was done. When an NG signal from the NG state signal output unit 8 is notified to the user, the NG signal may be notified via a sound or a dedicated display unit instead of being displayed on the image display unit 5.

The photographer notified that the photographed continuous image is NG gives an operation command via the operation control section 7 how to handle the target continuous image. The options for handling the photographer are, as in the case of non-NG, whether to save the image as a video in the data storage unit 3 or to discard it after judging that it has failed, or to play back and display it on the image display unit 5 and then save it. You can choose to do or discard.

The operation control section 7 gives parameters at the time of shooting, gives a start / off of shooting of the video camera, gives an instruction to display a continuous image or video in the continuous image memory section 2, the data storage section 3, etc. Receiving and executing all operation commands given to the photographing apparatus from the photographer, such as changing the reference for state discrimination of the discriminating section, giving an instruction when outputting an NG state signal, or giving an instruction when displaying an image. Control.

FIG. 2 is a block diagram showing a processing procedure of a photographing method with an NG warning function according to the embodiment of the present invention. Next, a shooting method with an NG warning function according to the present embodiment will be described with reference to FIG. First, the target object is photographed by the imaging means 11. The photographed continuous image 21 is temporarily held by the continuous image memory unit 12.
The imaging unit 11 may acquire general optical information and take an image, or may acquire other information. This is based on the imaging unit 1 described with reference to FIG. Next, in the information extracting means 14, the held continuous image 21 is analyzed, and various information 23 is extracted. Various information 23 extracted by the information extracting means 14
Are stored in the continuous image memory unit 12 together with the continuous image 21 captured by the imaging unit 11.

Using the extracted various information 23 as a parameter, the state discriminating means 16 determines whether or not the continuous image 21 is a video sufficient for editing work, that is, is suitable for editing (N
G) or not suitable for editing (not NG). If the state determination unit 16 determines that the image is not NG, the continuous image 21 temporarily stored in the continuous image memory unit 12 is stored in the video 2
2 is stored in the data storage unit 13. In this case, various information 23 is also stored in the data storage unit 13.
At this time, information indicating that the image has been stored may be notified to the photographer by, for example, reproducing and displaying the stored image 24 on the image display means 15. When the status is determined to be NG by the status determination unit 16, an NG signal 25 is transmitted to the photographer 32 via the NG status signal output unit 18. The transmission method may be output as image information, sound information, optical information other than the image, or other information.

Upon receiving the NG signal 25, the photographer 32 inputs some operation commands 30 via the operation control means 17. The input commands include a save command 26 for saving the continuous image 21 as a video as it is, a discard command 28 for discarding the continuous image 21 as it is, or a playback and display of a continuous image for once after confirming the content of the continuous image. There is a display command 27 to be performed. When the display command 27 is given, the continuous image 21 is displayed via the image display means 15. When the photographer 32 receives the NG signal 25, the photographer 32 may determine whether to save or discard based on the displayed result 31. This determination may be made by the same person as the photographer 32 or by another person. When the operation command 30 is not given despite the output of the NG signal 25, that is, the storage command 2
6, when the display command 27 and the discard command 28 are not given, the continuous image 21 may be stored as it is or may be discarded. Also, the information discriminating means 16 uses the various information 23
May be extracted in units of shots or only a part of the shots.

The image display means 15 displays all of the photographed image, the continuous image 21 in the continuous image memory means, and the video 24 stored in the data storage means 13.
The image display means 15 may have a dedicated display device, respectively, or may be simultaneously reproduced by switching by a certain method using a single device. Further, the display target is not all information, and may be a reduced image of each image, a thinned time information, or a slightly processed one including or not including audio information.

The timing for executing a series of means from the information extracting means 14 to the state discriminating means 16 and the NG state signal outputting means 18 is defined as a timing after an image ends in a shot unit corresponding to ON / OFF of the camera. But it's fine, even if you're shooting a shot. However, in the present embodiment, the operation command is given from the end of a shot until the next shot is taken. However, as an application, N
The video including the G signal may be stored, and an operation command may be individually given to only a portion where the NG signal exists at a later date. In this case, the various information 23 is also stored in the continuous image 21.
At the same time, since the data is stored in the data storage means 13, the editing work is facilitated by using the various information 23. Further, as an application of these systems, it may be determined whether or not to save the data by the operation command 30 even when there is no NG signal.

Next, the continuous image 21 is analyzed to obtain various information 2
3 for outputting information 3 and various information 2 obtained.
State determining means 1 for determining whether or not an image is NG based on the information 3
The determination method of No. 6 will be described below. First, what kind of state is assumed to be NG, in this embodiment,
(1) When the brightness is low and no information is displayed in black in a dark image, (2) When the brightness is too bright and no information is displayed in pure white, (3) When the image is severe due to severe camera shake If it violently vibrates, or if (4) the image cannot be identified due to sudden camera work, it is regarded as NG. However, the determination is not limited to satisfying any one of the above conditions, and may be determined from a plurality of conditions. State determination means 16
Then, the determination is performed based on a threshold value given as a parameter with information satisfying each condition.
Various conditions may be adopted, such as a threshold in the case of a single condition and a condition in which a plurality of conditions overlap.

Hereinafter, each of the above-mentioned conditions will be specifically described in order. FIG. 3 shows a continuous image 21 in the first case (when only dark and dark image information is obtained).
It is a figure showing an example of. In this case, the brightness distribution and the average of all the pixels in a part or several frames of the obtained continuous image 21 are calculated as the information 23. The state determination unit 16 determines that the image is entirely dark when the average brightness (E) is darker than a certain threshold (E1) and the variance (σ) is lower than a certain threshold (σ1). I will not output it because it has no meaning. Note that the number of frames for calculating the average value (E) and the variance (σ) of the brightness may be the entire image or may be extracted and determined at intervals of about one second.

FIG. 4 shows a continuous image 2 in the second case (only bright and bright image information is obtained).
FIG. 3 is a diagram showing an example of the first example. Also in this case, the brightness distribution and the average of all the pixels in a part or several frames of the obtained continuous image 21 are calculated as the information 23. The state determination unit 16 determines that the image is entirely bright when the average brightness (E) is brighter than a certain threshold (E2) and the variance (σ) is lower than a certain threshold (σ2). I will not output it because it has no meaning. In this case as well, the number of frames for calculating the average value (E) and the variance (σ) of the brightness may be the entire image or may be extracted and determined at intervals of about one second. In the first and second cases, the color of the pixel may be calculated as information, and if the image is a single color, the image may not be output because it is meaningless to edit.

FIG. 5 is a diagram showing an example of the continuous image 21 in the third (third) case (when camera shake has occurred). When the continuous image 21 is blurred, the entire image is slightly blurred.
As shown at 57, the content of the image is significantly blurred.
To detect this, a local motion vector between two frames is calculated. In order to simplify the processing, the motion vector is calculated in the present embodiment by dividing the image into 8 × 8 blocks and dividing the block area on the adjacent image as shown at 61 in FIG. A search is performed by changing a position within a certain range, and a movement position when the sum of absolute values of brightness differences between pixels constituting the block is the smallest and becomes smaller than a certain threshold is calculated as a motion vector. . The distribution of the motion vector is calculated based on the correspondence between the blocks. In calculating the motion vector, the brightness information of each pixel is used for block matching. However, the block size may be variable, or may be calculated for each pixel. In calculating the motion vector, color information of each pixel may be used for block matching.

In the case of camera shake, it is known that motion changes having a large moving distance appear in various directions in a short time. Therefore, it is first determined whether or not the camera work is instantaneously performed in a certain direction, and then it is determined whether or not such an instantaneous movement occurs in a different direction in several frames. First, in order to determine if such camerawork was taking place momentarily,
A vector sum 52 of motion vectors in the spatial direction and an average 53 of vector sizes are calculated for each frame. The fact that the camera work is instantaneously performed in a certain direction is only when the vector size average 53 is large and the vector sum 52 is large. Therefore, each threshold is given to detect that there is momentary movement.

Next, it is determined whether or not a frame having an instantaneous motion has continued for a certain length or more. In the present embodiment, the number of instantaneous movements per unit frame is calculated and compared with a given threshold. If a value larger than the threshold value is calculated, it means that the motion change is constantly occurring in the number of unit frames. In the case of camera shake, since the direction of such motion appears randomly, the sum over time of the vector sum for each frame is calculated, and if the size is smaller than a given threshold, the direction varies. Since it seems to have changed, it is determined that there is camera shake. If there is camera work, the image content will gradually move in the same direction of the camera work, so the motion vector of each frame must appear in the same direction in the same size and be continuous in time. become. Therefore, since the condition is completely different from the above-mentioned condition, it can be determined.

FIG. 7 is a diagram showing an example of the continuous image 21 in the fourth (fourth) case (when sudden camera work occurs). In the case of rapid camera work, 7 in FIG.
As in 1,72, the content of the image changes at a speed exceeding the viewer's prediction. At this time, the search for the motion vector used in the third case is not successful. That is, if the camera is stationary, many blocks with the motion vector 0 appear, and if the camera works slowly, it should be possible to track. Therefore, if there are many blocks where the search for the motion vector does not go well,
Suppose sudden camera work occurs.

The four conditions for NG have been described above. In addition to the above conditions, when a sudden change occurs in the same shot, such as when an object suddenly jumps out in front of the video camera, Generally, it is often NG.
Also in such a case, it is possible to detect a sudden change and calculate NG by applying the method under the above-described third and fourth conditions. Further, it can be easily inferred that there are many NG discriminating conditions other than the above embodiment.

The photographing method with an NG warning function according to the present embodiment can be executed by software processing of a general-purpose computer having an image pickup unit. In this case,
The shooting method with the NG warning function according to the present embodiment includes, for example,
A central processing unit (CPU) 91 shown in FIG.
This is performed by executing the photographing program read in. FIG. 8 is a block diagram showing a schematic configuration of computer hardware. In FIG.
Is a central processing unit (CPU), 92 is a main memory, 93 is a display, 94 is an input device such as a keyboard and a mouse,
95 is an auxiliary storage device such as a magnetic disk, 96 is an imaging device, and 97 is a bus line. The photographing program is supplied from, for example, a CD-ROM or the like, and is stored in the auxiliary storage device 95.

As described above, the invention made by the present inventor is:
Although a specific description has been given based on the above-described embodiment, the present invention is not limited to the above-described embodiment, and it is needless to say that various modifications can be made without departing from the gist of the invention.

[0040]

The effects obtained by typical ones of the inventions disclosed in the present application will be briefly described as follows.

(1) According to the present invention, in order to shoot a video which can be used for editing work, it is determined whether or not the target object is suitable for editing at the time of shooting. If an image that is not likely to be used for editing is shot, the user is notified that the shot image is not suitable for editing during or immediately after shooting the target object, so that a stable image can be used for sequential editing. It becomes possible to shoot a video that has been changed.

(2) According to the present invention, when it is determined that a captured image is not suitable for editing, the user is inquired as to whether or not the captured image may be held as a video, The user can decide whether to save the target video or discard the video and perform shooting again accordingly, and it is possible to shoot a video that can withstand the editing work with a high probability.

(3) According to the present invention, even a general user who has little shooting experience such as a camera work can easily shoot a video which can withstand editing.

(4) According to the present invention, since various information is stored in the data storage section together with the image, the editing operation when editing at a later date becomes easy.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a schematic configuration of a photographing apparatus with an NG warning function according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a processing procedure of a shooting method with an NG warning function according to the embodiment of the present invention.

FIG. 3 is a diagram illustrating an example of a continuous image when only dark and dark image information is obtained.

FIG. 4 is a diagram showing an example of a continuous image when only bright and bright image information is obtained.

FIG. 5 is a diagram illustrating an example of a continuous image in a case where camera shake has occurred.

FIG. 6 is a diagram for explaining calculation of a motion vector.

FIG. 7 is a diagram illustrating an example of a continuous image in a case where rapid camera work has occurred.

FIG. 8 is a block diagram illustrating a schematic configuration of hardware of a computer.

[Explanation of symbols]

REFERENCE SIGNS LIST 1 imaging unit 2 continuous image memory unit 3 data storage unit 4 information extraction unit 5 image display unit 6 state determination unit 7 operation control unit 8 NG state signal output unit 11 ...
Imaging means 12, continuous image memory means 13, data storage means 14, information extraction means 15, image display means 1,
6 state determination means, 17 operation control means, 18 NG signal output means, 21 continuous image, 22 video, 23 various information, 24 storage video, 25 NG signal, 26 storage instruction, 27 ... Display command, 28: Discard command, 29: Shooting command, 30: Operation command, 32: Photographer, 52: Vector sum, 53: Average vector, 55, 56, 57, 71, 7
2: part of the image, 91: central processing unit (CPU), 92
... Main memory, 93 ... Display, 94 ... Input device, 9
5: auxiliary storage device, 96: imaging device, 97: bus line.

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-10-304241 (JP, A) JP-A-7-120829 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) H04N 5/76-5/956 H04N 5/225-5/243

Claims (9)

(57) [Claims] An imaging device with an NG warning function for imaging an object and generating video information that is easy to use for editing, comprising: an imaging unit for imaging a target object; and an image temporarily captured by the imaging unit. continuous image memory unit and, in the image held in the continuous image memory unit, the brightness of holding the,
An information extraction unit that extracts various information including color, camera work, movement of a subject, and image blur; and from the various information extracted by the information extraction unit, an image held in the continuous image memory unit is suitable for editing. A state determination unit that determines whether or not the image is stored.If the state determination unit determines that the image held in the continuous image memory unit is not suitable for editing,
An NG state signal output unit for notifying that the image is not suitable for editing; a data storage unit for storing the image held in the continuous image memory unit; an image at the time of shooting; An imaging device with an NG warning function, comprising: an image display unit that displays an image or at least one of images stored in a data storage unit; and a control unit that controls each of the units. 2. The method according to claim 1, wherein the NG status signal output unit displays on the image display unit that the image is not suitable for editing, and notifies that the image is not suitable for editing. Item 2. An imaging device with an NG warning function according to Item 1. 3. The image processing apparatus according to claim 2, wherein the various information extracted by the information extracting unit is stored in the continuous image memory unit together with the image captured by the imaging unit, and is stored in the data storage unit. NG according to claim 1 or 2.
Shooting device with warning function. 4. A photographing method with an NG warning function for photographing an object and generating video information easy to use for editing, comprising: photographing a target object; temporarily retaining the photographed image; In the extracted image, various information including brightness, color, camera work, movement of the subject, and blur of the image is extracted.From the extracted various information, it is determined whether the held image is suitable for editing. Discriminating, when it is determined that the retained image is not suitable for editing, annunciating that the image is not suitable for editing, saving the retained image, image information at the time of shooting, retaining Displaying at least one of the stored image and the stored image.
G Shooting method with warning function. 5. The photographing method with an NG warning function according to claim 4, further comprising displaying that the image is not suitable for editing and notifying that the image is not suitable for editing. 6. The photographing method with an NG warning function according to claim 4, wherein the extracted various information is held and stored together with the photographed image. 7. A recording medium storing a photographing program for executing a photographing method with an NG warning function for photographing an object by a computer and generating video information easy to use for editing, wherein the photographing program is a computer. In, the image captured by the imaging device is temporarily stored in a memory, in the image stored in the memory, brightness, color, camera work, movement of the subject, various information including image blur, is extracted, From the extracted various information, it is determined whether the continuous image is suitable for editing.If it is determined that the image held in the memory is not suitable for editing, the image is suitable for editing. That the image is not stored, the image held in the memory is stored in the storage device, the image at the time of shooting, the image held in the memory, or the image stored in the storage device A recording medium on which a photographing program is recorded, wherein at least one of the images is displayed on a display unit. 8. The photographing program according to claim 7, wherein when notifying that the image is not suitable for editing, displaying that the image is not suitable for editing is displayed on the display unit. The recording medium on which it was recorded. 9. The apparatus according to claim 7, wherein the extracted various information is stored in the memory together with the image captured by the imaging unit, and is stored in the storage device. A recording medium on which a shooting program is recorded.