JP2021111868A - Imaging device and control method thereof, program, and storage medium - Google Patents

Abstract

To provide an imaging device capable of acquiring a user's favorite video without performing a special operation by a user.SOLUTION: An imaging device includes an imaging unit that captures a subject image, a detection unit that detects the shake of the imaging device, an image blur correction unit that corrects image blur on the imaging surface of the imaging unit due to the shake of the imaging device, an automatic photographing unit that causes the imaging unit to automatically take an image on the basis of the state of the imaging device, a recording unit that records video data automatically captured by the imaging unit and data in the image blur correction state by the image blur correction unit for the video data, and an automatic editing unit that automatically edits the video data on the basis of the image blur correction state data recorded by the recording unit.SELECTED DRAWING: Figure 5

Description

The present invention relates to an image pickup apparatus and a control method thereof.

There is known an automatic shooting camera in which the camera automatically determines the situation and continuously shoots, instead of shooting at an intended timing such as when the user releases the shutter (Patent Document 1). The automatic shooting camera is used in a state of being attached to the user's body with a strap or the like, and can automatically record the scene that the user sees in daily life as an image and leave the image of the user's preference.

However, since such a camera shoots automatically, if the user wears it and acts without worrying about the fact that the user is shooting, the image blur will be larger due to his / her own movement. May occur. Therefore, Patent Document 2 proposes a method of correcting image blur by changing the cutout position and the cutout size according to the amount of image blur during shooting.

Japanese Unexamined Patent Publication No. 2019-106694 Japanese Unexamined Patent Publication No. 2014-207507

However, in the prior art disclosed in Patent Documents 1 and 2 described above, when image blurring that exceeds the range of image blurring correction possible by the camera occurs, image blurring remains in the captured image. In some cases, a user-friendly image could not be obtained. In that case, the user had to manually edit and delete the frame of the part where the image blur of the image remained after shooting.

In addition, since shooting is performed with the camera attached to the user, if the user moves during shooting, the camera also moves and the scene or subject changes. Therefore, one image may include a plurality of scenes and subjects. In that case, the user had to manually edit the image after shooting in order to create an image for each scene or subject.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an imaging device capable of acquiring a user's favorite image without performing a special operation.

The imaging device according to the present invention is an imaging device, which includes an imaging means for capturing a subject image and an imaging device.
Based on the state of the image pickup apparatus, the detection means for detecting the shake of the image pickup device, the image shake correction means for correcting the image shake on the image pickup surface of the image pickup means due to the shake of the image pickup device, and the state of the image pickup device. Recording that records an automatic photographing means that causes an imaging means to automatically take a picture, video data automatically taken by the imaging means, and image blur correction state data by the image blur correction means with respect to the video data. It is characterized by including means and an automatic editing means for automatically editing the video data based on the image blur correction state data recorded by the recording means.

Further, the image pickup device according to the present invention is an image pickup device, which is an image pickup means for capturing a subject image, a detection means for detecting a shake of the image pickup device, and an image pickup of the image pickup means caused by the shake of the image pickup device. An inclination correction means for correcting the roll inclination on the surface, an automatic photographing means for causing the imaging means to automatically take an image based on the state of the imaging device, and an image automatically taken by the imaging means. Automatically edits the video data based on the recording means for recording the data and the data of the tilt correction state by the tilt correction means for the video data and the data of the tilt correction state recorded by the recording means. It is characterized by having an editing means.

Further, the image pickup device according to the present invention is an image pickup device, which is an image pickup means for capturing a subject image, a detection means for detecting the shake of the image pickup device, and an image pickup of the image pickup means caused by the shake of the image pickup device. Based on the image blur correction means for correcting the image blur on the surface, the tilt correction means for correcting the roll tilt on the image pickup surface of the image pickup means due to the shake of the image pickup device, and the state of the image pickup device. An automatic photographing means for causing the imaging means to automatically shoot, video data automatically captured by the imaging means, image blur correction state data by the image blur correction means for the video data, and the above. Based on the recording means for recording the data of the tilt correction state by the tilt correction means for the video data, the data of the image blur correction state recorded by the recording means, and the data of the tilt correction state, the video. It is characterized by including an automatic editing means for automatically editing data.

According to the present invention, it is possible to provide an imaging device capable of acquiring a user's favorite image without performing a special operation by the user.

The figure which shows typically the appearance of the camera which is 1st Embodiment of the image pickup apparatus of this invention. The block diagram which shows the structure of the camera of 1st Embodiment. The figure explaining the electronic blur correction. The flowchart explaining the operation of the 1st control part. A flowchart illustrating an automatic editing process. The figure explaining the deletion operation of the frame outside the blur correction range in automatic editing. The figure explaining the deletion operation of the frame outside the tilt correction range in automatic editing.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate explanations are omitted.

(First Embodiment)
<Camera configuration>
FIG. 1 is a diagram schematically showing the appearance of a camera according to a first embodiment of the imaging device of the present invention. The camera 101 shown in FIG. 1A is provided with a power switch, an operating member capable of operating the camera, and the like. A lens barrel 102 that integrally includes a photographing lens group and an image pickup element as an imaging optical system for imaging a subject image is movably attached to a fixed portion 103 of the camera 101. Specifically, the lens barrel 102 is attached to the fixed portion 103 via a tilt rotating unit 104 and a pan rotating unit 105, which are mechanisms that can be rotationally driven with respect to the fixed portion 103.

The tilt rotation unit 104 includes a motor drive mechanism capable of rotationally driving the lens barrel 102 in the pitch direction shown in FIG. 1 (b), and the pan rotation unit 105 rotates the lens barrel 102 in the yaw shown in FIG. 1 (b). It is equipped with a motor drive mechanism that can be rotationally driven in a direction. That is, the camera 101 has a mechanism for rotationally driving the lens barrel 102 in the biaxial direction. Each axis shown in FIG. 1B is defined with respect to the position of the fixed portion 103. The angular velocity meter 106 and the accelerometer 107 are arranged in the fixed portion 103 of the camera 101. Then, based on the output signals of the angular velocity meter 106 and the accelerometer 107, the vibration of the camera 101 is detected, and the tilt rotation unit 104 and the pan rotation unit 105 are rotationally driven to correct the vibration of the lens barrel 102. The tilt can be corrected.

FIG. 2 is a block diagram showing the overall configuration of the camera 101 of the present embodiment. In FIG. 2, the first control unit 224 includes, for example, a CPU (MPU), a memory (DRAM, SRAM), and the like. Then, according to the program stored in the non-volatile memory (EEPROM) 217, various processes are executed to control each block of the camera 101 and control the data transfer between the blocks. The non-volatile memory 217 is a memory that can be electrically erased and recorded, and stores constants, programs, and the like for the operation of the first control unit 224 as described above.

In FIG. 2, the zoom unit 201 includes a zoom lens that performs scaling (enlargement / reduction of an imaged subject image). The zoom drive control unit 202 drives and controls the zoom unit 201 and detects the focal length at that time. The focus unit 203 includes a focus lens that adjusts the focus (focus adjustment). The focus drive control unit 204 drives and controls the focus unit 203.

The image pickup unit 206 includes an image pickup element, receives light incident through each lens group, and outputs charge information corresponding to the amount of light to the image processing unit 207 as an analog image signal. The zoom unit 201, the focus unit 203, and the imaging unit 206 are arranged in the lens barrel 102.

The lens barrel rotation drive unit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to rotate the lens barrel 102 in the tilt direction and the pan direction.

The image processing unit 207 applies image processing such as distortion correction, white balance adjustment, and color interpolation processing to the digital image data obtained by A / D conversion of the analog image signal, and applies the applied digital image data to the digital image data. Output. The digital video data output from the video processing unit 207 is converted into a video recording format by the video recording unit 208, stored in the memory 216, or transmitted to the video output unit 218 described later.

The device tilt detection unit 209 and the device shake detection unit 210 include, for example, an angular velocity meter (gyro sensor) 106 that detects the angular velocity in the three-axis direction of the camera 101, and an accelerometer (accelerometer) that detects the acceleration in the three-axis direction of the device. 107 is provided. The device tilt detection unit 209 and the device shake detection unit 210 calculate the rotation angle of the camera 101, the amount of blurring of the camera 101, and the like based on the signals detected by the sensors. The first control unit 224 controls the lens barrel rotation drive unit 205 based on the calculation result, drives the tilt rotation unit 104 and the pan rotation unit 105, and optically causes an image on the imaging surface due to the shake of the camera 101. Correct blur and tilt. In addition, the first control unit 224 also performs electronic image blur correction that corrects image blur by moving the image cropping range based on the calculation result. The correction amount is calculated for each frame according to the calculation result, and a part of the image is cut out according to the correction amount.

Here, with reference to FIG. 3, an electronic image blur correction control for performing correction by cutting out a part of an image according to a correction amount for each frame will be described.

FIG. 3A is a diagram for explaining an image corrected by the electronic image blur correction performed by the first control unit 224. The roll tilt correction amount for performing rotation correction in the roll direction is calculated from the calculation result of the tilt correction angle. Image 301a shows a captured image input to the image processing unit 207. The image 302a shows a rotated image of the captured image 301a after the rotation process is performed based on the roll tilt correction amount. Further, in the electronic image blur correction, a process of cutting out an inscribed portion having the same aspect ratio as the captured image 301a as an output image 303a is performed from the rotated image 302a. By performing this cropping process for each frame of the video, electronic image blur correction in the tilt direction is realized.

Further, FIG. 3B is a diagram for explaining image blur correction in the vertical and horizontal directions (translation direction) of the image performed by the first control unit 224. In the translation direction correction, the image shift amount for correcting the angular runout in the pitch direction / yaw direction is calculated from the focal length information based on the runout angle and the zoom position information, and the image is cut out according to the image shift amount. Is done. Image 301b shows a captured image input to the image processing unit 207. The image 302b is an image corresponding to the angle of view cut out from the captured image 301b according to the amount of vertical and horizontal image shift. By performing this cropping process for each frame of the video, electronic image blur correction in the translation direction is realized.

By combining these optical corrections and electronic corrections, image blur correction and tilt correction are performed.

The voice input unit 214 acquires a voice signal around the camera 101 by a microphone provided in the camera 101, converts it into a digital voice signal, and transmits it to the voice processing unit 215. The voice processing unit 215 performs voice-related processing such as optimization processing of the input digital voice signal. Then, the voice signal processed by the voice processing unit 215 is transmitted to the memory 216 by the first control unit 224. The memory 216 temporarily stores the image signal and the audio signal obtained by the video processing unit 207 and the audio processing unit 215.

The automatic editing processing unit 220 reads out the video signal and audio signal temporarily stored in the memory 216, the rotation angle and the amount of deflection of the camera 101, cuts out, divides, deletes, etc. the video, and stores the video again in the memory 216. Let me.

The image processing unit 207 and the audio processing unit 215 read the image signal and the audio signal temporarily stored in the memory 216, encode the image signal, encode the audio signal, and the like, and perform the compressed image signal, the compressed audio signal, and the like. To generate. The first control unit 224 transmits these compressed image signals and compressed audio signals to the recording / reproducing unit 221.

The recording / playback unit 221 records the compressed image signal, the compressed audio signal, and other control data related to shooting on the recording medium 222, which are generated by the video processing unit 207 and the audio processing unit 215. When the audio signal is not compressed and encoded, the first control unit 224 transmits the audio signal generated by the audio processing unit 215 and the compressed image signal generated by the video processing unit 207 to the recording / playback unit 221. It is transmitted and recorded on the recording medium 222.

The recording medium 222 may be a recording medium built in the camera 101 or a removable recording medium, and can record various data such as a compressed image signal, a compressed audio signal, and an audio signal generated by the camera 101. Generally, a medium having a capacity larger than that of the non-volatile memory 217 is used as the recording medium 222. For example, the recording medium 222 includes all types of recording media such as hard disks, optical disks, magneto-optical disks, CD-Rs, DVD-Rs, magnetic tapes, non-volatile semiconductor memories, and flash memories.

The recording / reproducing unit 221 reads (reproduces) the compressed image signal, the compressed audio signal, the audio signal, various data, and the program recorded on the recording medium 222. Then, the first control unit 224 transmits the read compressed image signal and compressed audio signal to the video processing unit 207 and the audio processing unit 215. The video processing unit 207 and the audio processing unit 215 temporarily store the compressed image signal and the compressed audio signal in the memory 216, decode them according to a predetermined procedure, and transmit the decoded signals to the video output unit 218 and the audio output unit 219. do.

A plurality of microphones are arranged in the voice input unit 214, and the voice processing unit 215 can detect the direction of the sound with respect to the plane on which the plurality of microphones are installed, and is used for searching for a subject and automatic shooting described later. .. Further, the voice processing unit 215 detects a specific voice command. The voice command may be configured so that the user can register a specific voice in the camera in addition to some commands registered in advance. It also recognizes the sound scene at the time of imaging. In sound scene recognition, a sound scene is determined by a network learned by machine learning based on a large amount of voice data in advance. For example, a network for detecting a specific scene such as "cheering", "applause", or "speaking" is set in the voice processing unit 215, and the specific sound scene or specific scene is specified. Detect voice commands. When the voice processing unit 215 detects a specific sound scene or a specific voice command, it outputs a detection trigger signal to the first control unit 224 and the second control unit 212.

In addition to the first control unit 224 that controls the entire main system of the camera 101, a second control unit 212 that controls the power supply of the first control unit 224 is provided. The first power supply unit 211 and the second power supply unit 213 supply electric power for operating the first control unit 224 and the second control unit 212, respectively. By pressing the power button provided on the camera 101, power is first supplied to both the first control unit 224 and the second control unit 212. As will be described later, the first control unit 224 is the first power supply unit. It also controls to turn off its own power supply to 211. Even while the first control unit 224 is not operating, the second control unit 212 is operating, and information from the device tilt detection unit 209, the device shake detection unit 210, and the voice processing unit 215 is input. The second control unit 212 determines whether or not to start the first control unit 224 based on various input information, and when it is determined to start, the first power supply unit 211 tells the first control unit 224. Instruct to supply power to.

The audio output unit 219 outputs a preset audio pattern from a speaker built in the camera 101, for example, at the time of shooting. The LED control unit 225 lights the LED provided in the camera 101 based on a preset lighting pattern or blinking pattern, for example, at the time of shooting. The video output unit 218 is composed of, for example, a video output terminal, and outputs an image signal in order to display the video on a connected external display or the like. Further, the audio output unit 219 and the video output unit 218 may be one combined terminal, for example, a terminal such as an HDMI (registered trademark: High-Definition Multimedia Interface) terminal.

The communication unit 223 is a part that communicates between the camera 101 and an external device, and transmits or receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal, for example. In addition, it receives commands related to shooting such as shooting start and end commands, pan / tilt, and zoom drive, and drives the camera 101 based on instructions from an external device. In addition, information such as various parameters is transmitted and received between the camera 101 and the external device. The communication unit 223 includes, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a WirelessUSB (registered trademark), and a GPS receiver.

FIG. 4 is a flowchart illustrating an example of the operation undertaken by the first control unit 224 of the camera 101 in the present embodiment.

In S401, the image processing unit 207 is made to process the image signal captured by the imaging unit 206 to generate an image for subject recognition. A specific subject such as a person or an object is recognized with respect to the generated image. When recognizing a person, the face or human body of the subject is detected.

In the face detection process, a pattern for determining the face of a person is predetermined, and a portion of the captured image that matches this pattern can be detected as a face image of the person. At the same time, the reliability indicating the certainty of the subject's face is also calculated. The reliability is calculated from, for example, the size of the face area in the image, the degree of matching with the face pattern, and the like.

Furthermore, regarding object recognition, similarly, in the case of an animal such as a dog as an object included in the captured image, the object is recognized by determining whether or not it matches the pattern of the pre-registered animal. Can be done. As for the method of specific object recognition, a subject to be detected may be learned in advance using Convolutional Neural Networks (hereinafter abbreviated as CNN) or the like, and CNN may be applied to face recognition or object recognition.

There is also a method of extracting a feature subject by a method of using a histogram of hue, saturation, etc. in the captured image. In this case, the image of the subject captured within the shooting angle of view is divided into a plurality of sections based on the distribution of the histogram such as the hue and saturation, and the process of classifying the captured images for each section is executed. NS. For example, a histogram of a plurality of color components is created for the captured image, and the image is divided by the mountain-shaped distribution range. Then, the images captured are classified according to the regions belonging to the combination of the same sections, and the image region of the subject is recognized. By calculating the evaluation value for each image area of the recognized subject, the image area of the subject having the highest evaluation value can be determined as the main subject area.

By the above method, each subject information can be obtained from the imaging information, and the subject information, the scene information to be photographed, and the like can be determined.

In S402, the tilt correction amount and the image blur correction amount are calculated. Specifically, first, the absolute angle of the camera 101 is determined based on the angular velocity and acceleration information acquired by the device tilt detection unit 209 and the device shake detection unit 210. Make a calculation. Then, the image blur correction angle for moving the tilt rotation unit 104 and the pan rotation unit 105 is obtained in the angle direction that cancels the absolute angle, and is used as the tilt correction amount and the image blur correction amount.

In S403, pan / tilt drive is performed. The pan / tilt drive amount for tracking the subject recognized by S401 and the pan / tilt drive amount based on the image blur correction amount calculated in S402 are calculated, and the tilt rotation unit 104 and pan are calculated by the lens barrel rotation drive unit 205. The rotation unit 105 is driven and controlled, respectively.

In S404, the zoom unit 201 is controlled to drive the zoom. The zoom is driven according to the state of the subject recognized by S401. For example, when the subject is a human face, if the face on the image is too small to be smaller than the minimum detectable size, it cannot be detected and may be lost. In such a case, by zooming to the telephoto side, the size of the face on the image is controlled to be large. On the other hand, if the face on the image is too large, the subject tends to deviate from the angle of view due to the movement of the camera itself. In such a case, the size of the face on the screen is controlled to be smaller by zooming to the wide-angle side. By performing the zoom control in this way, it is possible to maintain a state suitable for tracking the subject.

In S405, it is determined whether or not to start automatic shooting. Here, it is determined whether or not to perform shooting by using the subject information, the scene determination result, the voice command, and the sound scene of the image recognition process performed in S401. For example, when a person or animal with a pattern registered in advance is detected by important subject recognition in a state where the scene is determined in S401 as a landscape, it is assumed that the important subject exists on the screen. Make a decision to do. In addition, when a sound scene in which cheers are raised is determined in a state where a plurality of people are detected, it is determined that shooting is performed assuming that some important event has occurred. The decision to perform this shooting may be made so that the appearance rate of an important subject in the image recognition process and the user's preference can be learned and changed. Further, the user may manually register an important subject and use it as a judgment for shooting.

If it is determined in S405 to start shooting, the process proceeds to S406 to determine the tilt correction range and the image blur correction range when shooting from now on. The tilt correction range and the image blur correction range are determined by changing the maximum range in which tilt correction and image blur correction can be performed according to the current rotation angle of the device, the amount of vibration of the device, and the shooting mode.

For example, in the case of a scene such as a landscape where tilt is a concern, emphasis is placed on tilt correction by allocating a large amount of the movable range for controlling tilt correction and image blur correction and the processing power of the processor to tilt correction. And set the image blur correction range narrow. If you are shooting a person and you are worried about blurring, distribute a large amount of correction range to image blurring correction and set the tilt correction range narrower.

When the tilt correction range and the image blur correction range are determined, automatic shooting is started in S407. At that time, the focus drive control unit 204 performs autofocus control. Further, exposure control is performed so that the subject has an appropriate brightness by using an aperture control unit (not shown), an image sensor gain control unit, and a shutter control unit. Further, during shooting, the tilt correction range and the correction state for the image blur correction range determined in S406 are also recorded in the memory 216. After shooting, the image processing unit 207 performs various image processes such as auto white balance processing, noise reduction processing, and gamma correction processing to generate an image.

In S408, the automatic editing processing unit 220 performs automatic editing processing using the video generated in S407 and the correction state. In the automatic editing process, a frame that exceeds the blur correction range or the tilt correction range is cut in the video. When cutting the frame, the position of the frame cut is changed by using the subject information and the scene determination information processed by the video processing unit 207 and the sound data processed by the audio processing unit 215. The automatic editing process will be described in detail with reference to FIGS. 5 and 6.

FIG. 5 is a flowchart illustrating an example of an operation undertaken by the automatic editing processing unit 220 of the camera 101 in the present embodiment.

<Explanation of processing when outside the image blur correction range>
In S501, the image generated by the image processing unit 207 and the image blur and tilt correction states (image blur correction state, tilt correction state) recorded in the memory 216 at the time of shooting the image were confirmed, and the determination was made in S406. Detects the frame range of the image outside the blur correction range that exceeds the correction range. However, when detecting the outside of the blur correction range, if the outside of the blur correction range and the inside of the correction range are repeated in a short time, the process may be such that the entire period is treated as outside the correction range. Further, if the time outside the blur correction range is very short, the threshold value of the correction range may be temporarily changed so as to be treated as within the correction range.

In S502, important subject information in the image is detected. For example, the scene determination information detects a period during which the same important subject continues to be detected, such as a period during which the same scene continues or a period during which the same dog is detected as an important subject.

In S503, the sound data in the video is detected. For example, the sound scene determination result is used to detect a period during which the same sound scene continues, such as a period during which cheers are rising, a period during which applause is being performed, and a period during which a voice is being emitted.

In S504, using the period in which the same important subject in S502 continues and the period in which the same sound scene in S503 continues, it is determined whether or not each period overlaps with the target frame to be deleted detected in S501. If there is a period that is the same as the frame that is symmetrical to be deleted, that frame is excluded from the deletion target. The method of excluding from the deletion target will be described with reference to FIG.

FIG. 6 is a diagram showing the image A600, the amount of blur 601, the subject detection period 602, and the face detection reliability 604 calculated in S401, with the passage of time as the horizontal axis.

The image A600 assumes a scene in which a person's face is photographed and the camera is moved to cause a large blur. With reference to the blur amount 601, the period during which a large blur to be deleted exceeds the correction range threshold value 603 determined in S406 is from the deletion target start position 606 to the last deletion target end position 607 of the video. .. However, in the subject detection period 602 detected in S502, face detection as an important subject is detected up to the face detection end position 609 (subject detection state). Further, if the face detection reliability 604 is higher than the face recognition threshold 605, it can be recognized as a face, but since the reliability decreases at the face detection reliability decrease position 608, it can be detected as a face thereafter, but the reliability decreases due to blurring. As a result, the image becomes unfavorable. Therefore, since the period from the deletion target start position 606 to the face detection reliability reduction position 608 is a period in which the face can be recognized even if the blur is large, it is considered that the period is a useful image, and the face detection reliability is determined from the deletion target start position 606. The process of removing the lowered position up to 608 from the deletion target is performed. Therefore, the actual deletion target is the period from the face detection reliability reduction position 608 to the last deletion target end position 607 of the video.

After performing such a determination process, in S505, the frames of the period determined in S504 and FIG. 6 are deleted, and the video is saved again.

As described above, in the first embodiment, it is possible to provide an image that suits the user's preference by deleting the portion of the captured image that exceeds the blur correction range.

(Second Embodiment)
In this second embodiment, the configuration and overall operation of the camera are the same as those in FIGS. 1, 2, and 4 described in the first embodiment, and thus the description thereof will be omitted. Hereinafter, the second embodiment will be described with reference to FIGS. 5 and 7 in a portion different from that of the first embodiment.

<Explanation of processing when outside the tilt correction range>
In the present embodiment, in S501 of FIG. 5, the image generated by the image processing unit 207 and the correction state of blur and inclination recorded in the memory 216 at the time of shooting the image are confirmed, and the inclination correction determined in S406 is performed. Detects video frames that exceed the correction range. Similar to the blur correction in the first embodiment, when the outside of the correction range of the tilt correction and the inside of the correction range are repeated in a short time, the process may be such that the entire period is treated as out of the correction range. Further, when the outside of the correction range is a very short time, the threshold value of the correction range may be temporarily changed so as to be treated as being within the correction range.

In S502, important subject information in the image is detected. For example, the scene determination information detects a period during which the same important subject continues to be detected, such as a period during which the same scene continues or a period during which the same dog is detected as an important subject.

In S503, the sound data in the video is detected. For example, the sound scene determination result is used to detect a period during which the same sound scene continues, such as a period during which cheers are rising, a period during which applause is being performed, and a period during which a voice is being emitted.

In S504, using the period in which the same important subject in S502 continues and the period in which the same sound scene in S503 continues, it is determined whether or not each period overlaps with the target frame to be deleted detected in S501. If there is a period that is the same as the frame that is symmetrical to be deleted, that frame is excluded from the deletion target. The method of excluding from the deletion target will be described with reference to FIG.

FIG. 7 is a diagram showing the image B 700, the tilt amount 701, the sound scene detection period 702, the subject detection period 703, and the face size 710 detected as an important subject, with the passage of time as the horizontal axis. In the video B700, a person was photographed, but the person moved away and the camera was moved to shoot a person of another subject and tilted greatly, and then the person approached while speaking out. It is assumed that the picture was taken.

In the tilt amount 701, the period during which a large tilt to be deleted exceeds the correction range threshold value 704 determined in S406 is from the deletion start position 705 to the deletion end position 706. However, in the subject detection period 703 detected in S502, the detection of the first face detected as the subject exists up to the face detection end position 707. Further, when the detected first face size is less than the face detection size threshold value 710, it can be recognized as a face, but it is small and unsuitable for shooting. Therefore, from the deletion start position 705 to the detection size threshold position 713 of the first face, the face can be recognized as a face although the inclination is large, which is a size suitable for shooting. Therefore, the area between the deletion start position 705 and the first face detection size threshold position 713 is regarded as a useful image even if the inclination is large, and the deletion start position is set from the position 705 to the first face detection size threshold position 713. Change to.

Further, in the subject detection period 703, the second face detection is started at the face detection start position 708. Further, the state in which the voice is generated is detected as the sound scene detection start position 709 even in the sound scene detection period 702. However, since the detection size of the second face gradually increases, the face size suitable for shooting cannot be obtained until after the detection size threshold position 714 of the second face. Therefore, after the second face detection size threshold position 714, the face size is suitable for shooting, and the state in which a voice is generated is detected, and a video containing useful sound scenes and scenes. Is determined to be. Therefore, the deletion end position is changed from the position 706 to the detection size threshold position 714 of the second face.

After performing such a determination process, in S505, the frames of the period determined in S504 and FIG. 7 are deleted, and one image is saved again as a plurality of images divided for each scene.

As described above, in the second embodiment, it is possible to provide an image that suits the user's preference by deleting the portion that exceeds the inclination correction range of the captured image.

Although the case where the tilt correction range is exceeded is described in the present embodiment, both the case where the tilt correction range is exceeded and the case where the tilt correction range is exceeded as described in the first embodiment are described. It may be used to determine.

As described above, according to the first and second embodiments described above, the user manually edits the recorded video even if the captured video has uncorrectable blurring or tilting. You can reduce the trouble. Further, when a scene or a subject changes before and after an image blur or tilt that cannot be corrected, it can be divided into an image including the same subject or only the same scene.

(Other embodiments)
The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

101: Camera, 104: Tilt rotation unit, 105: Pan rotation unit, 206: Imaging unit, 207: Video processing unit, 209: Device tilt detection unit, 210: Device shake detection unit, 212: Second control unit, 224: 1st control unit

Claims (20)

It is an image pickup device
An imaging means for capturing a subject image and
A detection means for detecting the runout of the image pickup apparatus and
An image blur correction means for correcting image blur on the image pickup surface of the image pickup means due to the shake of the image pickup apparatus, and an image blur correction means for correcting the image blur on the image pickup surface of the image pickup means.
An automatic photographing means for causing the imaging means to automatically take an image based on the state of the image pickup device, and
A recording means for recording video data automatically captured by the imaging means and data in an image blur correction state by the image blur correction means for the video data.
An automatic editing means that automatically edits the video data based on the image blur correction state data recorded by the recording means, and
An imaging device characterized by comprising. Claim 1 is characterized in that the image blur correction means determines an image blur correction range based on a state of vibration of the image pickup apparatus detected by the detection means before shooting by the automatic photographing means. The imaging apparatus according to the above. When the image blur correction state of each frame of the video data captured by the imaging means exceeds the image blur correction range, the automatic editing means deletes the frame exceeding the image blur correction range from the video data. The imaging device according to claim 2, wherein the image pickup apparatus is used. The recording means further records sound data at the time of imaging of the video data captured by the imaging means, and the automatic editing means is based on the sound data in addition to the data in the image blur correction state. The imaging device according to claim 3, wherein the video data is automatically edited. The imaging device according to claim 4, wherein the automatic editing means changes the position of the frame to be deleted, which is determined based on the data of the image blur correction state, based on the sound data. A subject detecting means for detecting a subject from the video data captured by the imaging means is further provided, and the automatic editing means is based on the subject detecting state of the subject detecting means in addition to the data of the image blur correction state. The imaging device according to claim 3, wherein the video data is automatically edited. The imaging device according to claim 6, wherein the automatic editing means changes the position of the frame to be deleted, which is determined based on the data of the image blur correction state, based on the subject detection state. It is an image pickup device
An imaging means for capturing a subject image and
A detection means for detecting the runout of the image pickup apparatus and
An inclination correction means for correcting the roll inclination of the image pickup means on the image pickup surface due to the runout of the image pickup apparatus, and an inclination correction means.
An automatic photographing means for causing the imaging means to automatically take an image based on the state of the image pickup device, and
A recording means for recording video data automatically captured by the imaging means and data in a tilt correction state by the tilt correction means for the video data.
An automatic editing means that automatically edits the video data based on the tilt correction state data recorded by the recording means, and
An imaging device characterized by comprising. The eighth aspect of the present invention is characterized in that the tilt correction means determines a tilt correction range based on a state of vibration of the imaging device detected by the detection means before shooting by the automatic photographing means. Imaging device. When the tilt correction state of each frame of the video data captured by the imaging means exceeds the tilt correction range, the automatic editing means deletes the frame exceeding the tilt correction range from the video data. The imaging apparatus according to claim 9. The recording means further records sound data at the time of imaging of the video data captured by the imaging means, and the automatic editing means further records the video based on the sound data in addition to the data in the tilt correction state. The imaging device according to claim 10, wherein the data is automatically edited. The imaging device according to claim 11, wherein the automatic editing means changes the position of the frame to be deleted, which is determined based on the data of the tilt correction state, based on the sound data. The subject detecting means for detecting a subject from the video data captured by the imaging means is further provided, and the automatic editing means is based on the subject detection state of the subject detecting means in addition to the data of the tilt correction state. The imaging device according to claim 10, wherein the video data is automatically edited. The imaging device according to claim 13, wherein the automatic editing means changes the position of the frame to be deleted, which is determined based on the data of the tilt correction state, based on the subject detection state. It is an image pickup device
An imaging means for capturing a subject image and
A detection means for detecting the runout of the image pickup apparatus and
An image blur correction means for correcting image blur on the image pickup surface of the image pickup means due to the shake of the image pickup apparatus, and an image blur correction means for correcting the image blur on the image pickup surface of the image pickup means.
An inclination correction means for correcting the roll inclination of the image pickup means on the image pickup surface due to the runout of the image pickup apparatus, and an inclination correction means.
An automatic photographing means for causing the imaging means to automatically take an image based on the state of the image pickup device, and
The video data automatically captured by the imaging means, the image blur correction state data of the image blur correction means for the video data, and the tilt correction state data of the tilt correction means for the video data are recorded. Recording means and
An automatic editing means for automatically editing the video data based on the image blur correction state data recorded by the recording means and the tilt correction state data.
An imaging device characterized by comprising. A method of controlling an imaging device including an imaging means for capturing a subject image.
A detection step for detecting the runout of the image pickup apparatus and
An image blur correction step for correcting image blur on the imaging surface of the imaging means due to the shake of the imaging device, and an image blur correction step.
An automatic imaging step of causing the imaging means to automatically perform imaging based on the state of the imaging device, and
A recording step of recording video data automatically captured by the imaging means and data of an image blur correction state of the video data by the image blur correction step.
An automatic editing step of automatically editing the video data based on the image blur correction state data recorded in the recording step, and
A method for controlling an imaging device, which comprises. A method of controlling an imaging device including an imaging means for capturing a subject image.
A detection step for detecting the runout of the image pickup apparatus and
A tilt correction step for correcting the roll tilt on the imaging surface of the imaging means due to the runout of the imaging device, and a tilt correction step.
An automatic imaging step of causing the imaging means to automatically perform imaging based on the state of the imaging device, and
A recording step of recording video data automatically captured by the imaging means and data of a tilt correction state of the video data by the tilt correction step.
An automatic editing step of automatically editing the video data based on the tilt correction state data recorded in the recording step, and
A method for controlling an imaging device, which comprises. A method of controlling an imaging device including an imaging means for capturing a subject image.
A detection step for detecting the runout of the image pickup apparatus and
An image blur correction step for correcting image blur on the imaging surface of the imaging means due to the shake of the imaging device, and an image blur correction step.
A tilt correction step for correcting the roll tilt on the imaging surface of the imaging means due to the runout of the imaging device, and a tilt correction step.
An automatic imaging step of causing the imaging means to automatically perform imaging based on the state of the imaging device, and
The video data automatically captured by the imaging means, the data of the image blur correction state of the video data by the image blur correction step, and the data of the tilt correction state of the video data by the tilt correction step are recorded. Recording process and
An automatic editing step of automatically editing the video data based on the image blur correction state data recorded in the recording step and the tilt correction state data.
A method for controlling an imaging device, which comprises. A program for causing a computer to execute each step of the control method according to any one of claims 16 to 18. A computer-readable storage medium that stores a program for causing a computer to execute each step of the control method according to any one of claims 16 to 18.

Images