JP2015139100A - Image processing device, imaging apparatus, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress the degradation of the image quality of an image after correction processing is performed on the image obtained by imaging, even when a large vibration occurs due to the movement of a user.SOLUTION: An image processing device 50 includes image data storage section 15 storing imaged image data, a vibration information storage section 17 storing vibration information when imaging image data, a cutout zoom ratio/correction parameter determination section 19 determining the size of the cutout area of the imaged image data, according to the vibration information stored in the vibration information storage section 17, and a cutout image generation section 21 generating the cutout image data of the image data stored in the image data storage section 15, according to the determined size of the cutout area.

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, and an imaging processing method that process image data obtained by imaging.

  2. Description of the Related Art In recent years, for example, a camera device that is called a wearable camera or an action camera and captures and records an image of a situation around a user at all times or in response to a user input operation is known. In such a camera device, for example, a fish-eye lens or a lens having a wide angle of view is used as an optical system in order to pick up an image in a wide angle range. The wearable camera or the action camera is often used while being worn by a user. Therefore, the wearable camera or the action camera often vibrates according to the movement of the user during imaging.

  Here, as a prior art that outputs a perspective projection image obtained by performing a shake correction process on a fisheye image captured using a fisheye lens, for example, an imaging apparatus disclosed in Patent Document 1 is known.

  The imaging apparatus shown in Patent Literature 1 performs perspective projection conversion based on vibration information obtained by detecting a motion vector by a template matching method when performing shake correction processing on a fisheye image obtained by imaging using a fisheye lens. Shift the target area. Thereby, the imaging device shown in patent document 1 can reduce the amount of calculation at the time of converting a fish-eye image into a perspective projection image.

JP 2005-252625 A

  Here, even when the configuration of the imaging device of Patent Document 1 is applied to the above-described wearable camera or action camera that is used by being worn on the user's body, the imaging device is greatly shaken by the movement of the user In this case, only a region to be converted from a fisheye image obtained by imaging to a perspective projection image is shifted, and the size (size) of the region to be converted itself is constant. For this reason, depending on how the size of the region itself to be converted is taken, there is a problem in that the phenomenon of image deterioration due to a decrease in resolution and shaking is caused.

  In order to solve the above-described conventional problems, the present invention performs adaptive image correction according to the magnitude of the user's shaking, thereby suppressing the viewer's uncomfortable feeling and providing optimal video quality. An object of the present invention is to provide an image processing apparatus, an imaging apparatus, and an image processing method.

  The present invention relates to an image data storage unit that stores captured image data, a vibration information storage unit that stores vibration information at the time of imaging of the image data, and the vibration information stored in the vibration information storage unit. A cutout area determination unit that determines the size of the cutout area of the captured image data, and the image data storage unit stores the cutout area according to the size of the cutout area determined by the cutout area determination unit. And a cutout image generation unit that generates cutout image data of the image data.

  In this configuration, the cutout region determination unit variably determines the size of the cutout region of the captured image data in accordance with the vibration information stored in the vibration information storage unit. The cutout image generation unit generates cutout image data of the image data according to the size of the cutout region determined by the cutout region determination unit.

  As a result, the image processing apparatus reduces the size of the cut-out area during correction processing (for example, cut-out processing) of captured image data when a large shake occurs according to the movement of the user. Even in the case of occurrence of cutout, cut-out image data in which the disturbance of the image due to shaking is suppressed can be obtained. In addition, when a small shake occurs according to the user's movement, the image processing apparatus increases the size of the cutout area during the correction processing of the captured image data. Cut-out image data with high image quality can be obtained. That is, the image processing apparatus can perform adaptive image correction according to the magnitude of the user's shake, thereby suppressing the viewer's uncomfortable feeling and providing the optimum video quality.

  Further, according to the present invention, the cutout region determination unit calculates a shake correction value indicating the center of the cutout region according to the vibration information stored in the vibration information storage unit, and the calculated shake correction value The image processing apparatus determines a ratio of the size of the cut-out image data to the size of the captured image data.

  In this configuration, the cutout region determination unit calculates a shake correction value (yaw, pitch) indicating the center (position) of the cutout region according to the vibration information stored in the vibration information storage unit, and the calculated shake. In accordance with the correction value (yaw, pitch), the ratio of the size of the clipped image data (zoom rate) when the size of the captured image data is set to 100% is determined.

  As a result, the image processing apparatus uses the shake correction value (yaw, pitch) indicating the center position of the captured image data, and the size of the cut-out image data required during the correction processing of the size of the captured image data. Can be easily determined.

  Further, according to the present invention, the cutout region determination unit calculates a shake correction value indicating the center of the cutout region according to the vibration information stored in the vibration information storage unit, and the calculated shake correction value The image processing apparatus corrects the centering coefficient for each frame of the captured image data according to the above.

  In this configuration, the cutout region determination unit calculates a shake correction value (yaw, pitch) indicating the center (position) of the cutout region according to the vibration information stored in the vibration information storage unit, and the calculated shake. In accordance with the correction value (yaw, pitch), the centering coefficient for each frame necessary for the correction processing of the captured image data is corrected.

  As a result, the image processing apparatus uses the shake correction value (yaw, pitch) indicating the center position of the imaged image data to use the centering coefficient for each frame necessary for the correction process of the size of the imaged image data. Can be easily corrected.

  Further, the present invention provides an image processing in which the cutout region determination unit changes the size of the cutout region for each frame so that the size of the predetermined cutout region is maintained over a plurality of frames of the image data. Device.

  In this configuration, the cutout region determination unit sets the size of the cutout region for each frame so that the size of the cutout image data after the correction processing is the predetermined cutout region size over a plurality of image data frames. change.

  As a result, the image processing apparatus, when correcting the image data using the vibration information or the shake correction value (yaw, pitch), for example, the ratio of the size of the cut-out image data to the image data obtained by imaging is 80%. It can be changed to decrease by 5% for each frame. Therefore, the image processing apparatus, for example, generates an eye that accompanies a sudden change in the angle of view of the cut-out image data, such as the ratio of the size of the cut-out image data to the image data obtained by imaging changing from 80% to 60%. Can avoid a sense of incongruity.

  In addition, the present invention is an image processing apparatus further including a transmission unit that transmits the cut-out image data generated by the cut-out image generation unit to an external device connected via a network.

  In this configuration, the transmission unit transmits the cut-out image data obtained by the correction process on the captured image data to an external device connected to the image processing apparatus via the network.

  Thereby, the image processing apparatus can transmit the cut-out image data obtained by the correction process on the captured image data to the external device.

  In addition, the present invention provides an imaging unit that captures an image, a vibration sensor that detects vibration information at the time of capturing the image, and an image captured by the imaging unit according to the vibration information detected by the vibration sensor. A cutout area determination unit that determines the size of a cutout area of the image data of the video, and the image data captured by the imaging unit according to the size of the cutout area determined by the cutout area determination unit And a cutout image generation unit that generates cutout image data.

  In this configuration, the cutout region determining unit determines the size of the cutout region of the image data of the captured video according to the vibration information detected by the vibration sensor when the video is captured by the imaging unit. The cutout image generation unit generates cutout image data of the image data of the captured video in accordance with the size of the cutout region determined by the cutout region determination unit.

  As a result, the imaging device reduces the size of the cut-out area during correction processing (for example, cut-out processing) of the captured image data when a large shake occurs in accordance with the movement of the user. Even when it occurs, cut-out image data in which the disturbance of the image due to the shaking is suppressed is obtained. In addition, when a small shake occurs according to the movement of the user, the imaging device increases the size of the cutout area during the correction processing of the captured image data. High cutout image data can be obtained. That is, the imaging apparatus can provide an optimal video quality by suppressing the viewer's uncomfortable feeling by performing adaptive image correction according to the magnitude of the user's shake.

  Further, the present invention is an image processing method in an image processing apparatus having captured image data and vibration information at the time of capturing the image data, wherein the image data cut-out region is captured according to the vibration information And a step of generating cutout image data of the image data in accordance with the determined size of the cutout region.

  In this method, the image processing apparatus variably determines the size of the cutout area of the captured image data according to the vibration information, and the cutout image of the image data according to the determined size of the cutout area. Generate data.

  As a result, the image processing apparatus reduces the size of the cut-out area during correction processing (for example, cut-out processing) of captured image data when a large shake occurs according to the movement of the user. Even in the case of occurrence of cutout, cut-out image data in which the disturbance of the image due to shaking is suppressed can be obtained. In addition, when a small shake occurs according to the user's movement, the image processing apparatus increases the size of the cutout area during the correction processing of the captured image data. Cut-out image data with high image quality can be obtained. That is, the image processing apparatus can provide an optimal video quality by suppressing an uncomfortable feeling of the viewer by performing adaptive image correction according to the magnitude of the user's shake.

  ADVANTAGE OF THE INVENTION According to this invention, by performing adaptive image correction according to the magnitude | size of a user's shaking, a viewer's discomfort can be suppressed and optimal video quality can be provided.

(A) Explanatory drawing which shows an example of the cutout area | region of the original image data determined by the imaging device 100 of this embodiment in case a shake is small, and the cutout image data obtained by the correction process, (B) In the case where a shake is large Explanatory drawing which shows an example of the cutout area | region of the original image data which the imaging device 100 of this embodiment determines, and the cutout image data obtained by the correction process 1 is a block diagram showing in detail an internal configuration of an image processing apparatus or an imaging apparatus according to the present embodiment. (A) External appearance perspective view of the imaging device of this embodiment connected with the battery part by the cable, (B) The figure which shows an example of a mode that the imaging device of this embodiment was mounted | worn by the person The figure which shows an example of the table which shows the relationship between a shake correction value, a vibration level, a zoom rate, and a centering coefficient. The flowchart which shows an example of the detailed operation | movement procedure of the imaging device of this embodiment. (A)-(F) The figure which shows an example of the cut-out area | region of cut-out image data according to a shake correction value

  Hereinafter, an embodiment of an image processing device, an imaging device, and an image processing method according to the present invention (hereinafter referred to as “the present embodiment”) will be described with reference to the drawings. Hereinafter, the imaging apparatus according to the present invention will be exemplified to mainly describe the imaging apparatus according to the present embodiment, and the image processing apparatus and the image processing method according to the present invention will be described as necessary. The present invention is not limited to the invention of the apparatus category of the image processing apparatus or the imaging apparatus, and may be the invention of the method category of the image processing method that defines the operation of the image processing apparatus or the imaging apparatus.

  The imaging apparatus according to the present embodiment is an action that is worn on a wearable camera or a vehicle such as a bicycle or a boat that a user can hang around his / her neck and record surrounding images while the power is on. Used as a camera. Moreover, the imaging device of the present embodiment may be used as an in-vehicle camera mounted on a moving body (for example, a vehicle).

(Overview of the operation of the imaging device)
First, an outline of the operation of the imaging apparatus 100 of the present embodiment will be described with reference to FIGS. 1 (A) and 1 (B). FIG. 1A is an explanatory diagram illustrating an example of a cut-out area of original image data determined by the imaging apparatus 100 according to the present embodiment and cut-out image data obtained by correction processing when the shake is small. FIG. 1B is an explanatory diagram illustrating an example of a cut-out area of original image data determined by the imaging apparatus 100 according to the present embodiment and cut-out image data obtained by correction processing when shaking is large.

  As shown in FIG. 1A, for example, when a shake generated by a user's movement such as walking is small, the imaging apparatus 100 uses image data (hereinafter referred to as “original image data”) of a video obtained by imaging. ) Correction processing (for example, cutout processing), a large ratio (for example, 80%) of the range of the original image data is determined as the cutout region RNa, and cutout image data CNa corresponding to the cutout region RNa is generated and output. .

  That is, when the fluctuation is small, even if the influence of some fluctuation appears in the original image data, it is considered that there is little disturbance in the image due to the fluctuation of the cut-out image data. The correction process is not prioritized, and the cutout area (cutout angle of view) becomes large (see FIG. 1A). Therefore, the index indicating the overall visibility of the clipped image data CNa is still high depending on the resolution of the clipped image data CNa, that is, the size of the clipped image data CNa, and some fluctuation remains in the clipped image data CNa. There are few things.

  Although details will be described later, the imaging apparatus 100 uses a lens LN having a wide angle of view. For this reason, the imaging apparatus 100 needs to convert the original image data obtained by imaging into a data format suitable for storage in the imaging apparatus 100 itself or transmission to an external device, and thus the original image data itself is used. However, the original image data is corrected (for example, cutout processing, distortion correction processing, and centering processing) to generate cutout image data in which all or part of the original image data is cut out. The centering process is a process for correcting the center position of the original image data, but may be included as part of the distortion correction process.

  On the other hand, as shown in FIG. 1B, for example, when the shaking generated by the movement of the user or the like is large, the imaging apparatus 100 performs original image data correction processing (for example, clipping processing) in the original image data. Is selected as a cutout area RNb, and cutout image data CNb corresponding to the cutout area RNb is generated and output.

  That is, when the shake is large, it is considered that the disturbance of the image due to the shake is large. Therefore, in the correction process in the imaging apparatus 100, the correction process of the shake itself is prioritized, and the cutout area (cutout angle of view) is reduced. Therefore, the cutout image data CNb shown in FIG. 1B has a smaller image size than the cutout image data CNa shown in FIG. That is, the resolution of the cut-out image data CNb is deteriorated as compared with the resolution of the cut-out image data CNa, but the image distortion due to the shake is improved as compared with the original image data by the shake correction.

(Configuration of imaging device)
FIG. 2 is a block diagram showing in detail the internal configuration of the image processing apparatus 50 or the imaging apparatus 100 of the present embodiment. 2 includes a lens LN, an image sensor 11, an image data storage unit 13, a vibration sensor 15, a vibration information storage unit 17, and a cut-out zoom ratio / correction parameter determination unit 19. A cut-out image generation unit 21, a video encoder 23, a storage medium 25, and a network interface 27.

  The image data storage unit 13, the vibration information storage unit 17, the cutout zoom ratio / correction parameter determination unit 19, the cutout image generation unit 21, the video encoder 23, the storage medium 25, and the network interface 27 are used. An image processing apparatus 50 according to this embodiment is configured. As a result, the image processing apparatus 50 has the same configuration as the effects of the configuration of the imaging apparatus 100 by including the configuration excluding the lens LN, the image sensor 11, and the vibration sensor 15 among the configurations of the respective units of the imaging apparatus 100. Is obtained.

  The lens LN collects light (light rays) incident from the outside of the imaging device 100 and forms an image on a predetermined imaging surface of the image sensor 11. As the lens LN, a fisheye lens or a lens capable of obtaining a wide angle of view of, for example, 140 degrees or more is used.

  The image sensor 11 is a CCD (Charged-Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) solid-state image sensor, and converts an optical image formed on the imaging surface into an electrical signal. The lens LN and the image sensor 11 have a function as the imaging unit IM in the imaging device 100. The output of the image sensor 11 is input to, for example, a signal processing unit (not shown), and a frame of video image data defined by RGB (Red Green Blue) or YUV (luminance / color difference) recognizable by a person is generated. The

  The image data storage unit 13 is configured by using a semiconductor memory such as a DRAM (Dynamic Random Access Memory), for example, and stores image data of a video generated by a signal processing unit (not shown). The video image data stored in the image data storage unit 13 is read out by the cut-out image generation unit 21.

  The vibration sensor 15 is configured using, for example, an angular velocity sensor (gyro sensor) or an acceleration sensor, and detects a vibration state of the image capturing apparatus 100 that is generated in accordance with, for example, the movement of the user wearing the image capturing apparatus 100. As the vibration sensor 15, for example, an angular velocity sensor is used when detecting the shaking state of the imaging apparatus 100, and an acceleration sensor is used when detecting the tilt of the housing of the imaging apparatus 100.

  The vibration information storage unit 17 is configured by using a semiconductor memory such as a DRAM (Dynamic Random Access Memory), for example, and stores vibration information of the imaging device 100 detected by the vibration sensor 15. The vibration information of the imaging device 100 stored in the vibration information storage unit 17 is read out to the cutout zoom rate / correction parameter determination unit 19.

  The cutout zoom rate / correction parameter determination unit 19 as an example of the cutout region determination unit cuts out original image data imaged by the imaging unit IM in accordance with the vibration information of the imaging device 100 stored in the vibration information storage unit 17. The size of the area (zoom rate) and the correction parameters necessary for the correction process are determined.

  More specifically, the cut-out zoom ratio / correction parameter determination unit 19 is based on the vibration information of the imaging device 100 stored in the vibration information storage unit 17, for one frame of original image data captured by the imaging unit IM. A shake correction value (yaw, pitch) indicating the center position of the cutout area is calculated. Note that the calculation method of the shake correction value (yaw, pitch) in the cut-out zoom ratio / correction parameter determination unit 19 is a known technique, and thus the description thereof is omitted.

  The cut-out zoom ratio / correction parameter determination unit 19 categorizes the vibration level indicating the shaking state of one frame of the original image data imaged by the imaging unit IM based on the shake correction value (yaw, pitch).

  Here, the relationship between the shake correction value (yaw, pitch) and the vibration level indicating the shake state of one frame of the image data will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a table CT indicating the relationship among the shake correction value, the vibration level, the zoom rate, and the centering coefficient. The cut-out zoom ratio / correction parameter determination unit 19 determines a vibration level indicating the swing state of one frame of the original image data corresponding to the shake correction value (yaw, pitch) according to the contents of the table CT shown in FIG.

For example, the cut-out zoom ratio / correction parameter determination unit 19 calculates {| yaw | + | pitch |} / 2 using the shake correction value (yaw, pitch). The cut-out zoom ratio / correction parameter determination unit 19
(1) When {| yaw | + | pitch |} / 2 is 25 degrees or more, the vibration level indicating the shaking state of one frame of the original image data imaged by the imaging unit IM is determined as 5. ,
(2) When {| yaw | + | pitch |} / 2 is a 20 to 25 degree angle, the vibration level indicating the shaking state of one frame of the original image data imaged by the imaging unit IM is determined to be 4. And
(3) When {| yaw | + | pitch |} / 2 is 15 to 20 degrees or more, the vibration level indicating the shaking state of one frame of the original image data imaged by the imaging unit IM is 3. Decide
(4) When {| yaw | + | pitch |} / 2 is 10 to 15 degrees or more, the vibration level indicating the shaking state of one frame of the original image data imaged by the imaging unit IM is 2. Decide
(5) When {| yaw | + | pitch |} / 2 is 5 to 10 degrees or more, the vibration level indicating the shaking state of one frame of the original image data imaged by the imaging unit IM is 1. Decide
(6) When {| yaw | + | pitch |} / 2 is 0 to 5 degrees or more, the vibration level indicating the shaking state of one frame of the original image data imaged by the imaging unit IM is set to 0. decide.

  Further, the cut-out zoom ratio / correction parameter determination unit 19 performs original image data imaged by the imaging unit IM in accordance with the vibration level indicating the shaking state of one frame of the original image data in accordance with the contents of the table CT shown in FIG. The ratio (cutout zoom rate) between the size of one frame and the size of one frame of the cutout image data is determined. Thereby, the cut-out zoom ratio / correction parameter determination unit 19 is necessary at the time of correcting the size of the captured image data using the shake correction value (yaw, pitch) indicating the center position of the captured image data. The size of the cutout image data to be obtained can be easily determined.

For example, the cut-out zoom ratio / correction parameter determination unit 19
(1) When the vibration level indicating the shaking state of one frame of the original image data is 5, the size of one frame of the original image data captured by the imaging unit IM and the size of one frame of the cut-out image data The ratio (cutout zoom rate) is determined to be 60%,
(2) When the vibration level indicating the shaking state of one frame of the original image data is 4, the size of one frame of the original image data captured by the imaging unit IM and the size of one frame of the cut-out image data The ratio (cutout zoom rate) is determined as 65%,
(3) When the vibration level indicating the shaking state of one frame of the original image data is 3, the size of one frame of the original image data captured by the imaging unit IM and the size of one frame of the cut-out image data The ratio (cutout zoom ratio) is determined to be 70%,
(4) When the vibration level indicating the shaking state of one frame of the original image data is 2, the size of one frame of the original image data captured by the imaging unit IM and the size of one frame of the cut-out image data The ratio (cutout zoom rate) is determined to be 75%,
(5) When the vibration level indicating the shaking state of one frame of the original image data is 1, the size of one frame of the original image data captured by the imaging unit IM and the size of one frame of the cut-out image data The ratio (cutout zoom rate) is determined as 80%,
(6) When the vibration level indicating the shaking state of one frame of the original image data is 0, the size of one frame of the original image data captured by the imaging unit IM and the size of one frame of the cut-out image data The ratio (cutout zoom ratio) is determined to be 85%.

  Further, the cut-out zoom ratio / correction parameter determination unit 19 performs correction parameters necessary for correcting one frame of the original image data in accordance with the shake correction value (yaw, pitch) according to the contents of the table CT shown in FIG. (For example, the centering coefficient α) is corrected. The cut-out zoom rate / correction parameter determination unit 19 outputs information on the determined cut-out zoom rate and correction parameters to the cut-out image generation unit 21.

  Here, the centering coefficient α is a value determined in accordance with the distance between the center position of the cutout area indicated by the shake correction value (yaw, pitch) and the center position of the fisheye lens (center coordinates of the original image data) ( Coefficient), which is one of the correction parameters used in the correction process of the original image data. The centering coefficient α is calculated for each frame of the original image data by the cut-out image generation unit 21 described later.

  When the distance between the center position of the cutout area indicated by the shake correction value (yaw, pitch) and the center coordinates of the original image data is large, the cutout zoom ratio / correction parameter determination unit 19 decreases the centering coefficient α. For example, the centering coefficient α is corrected to a value away from 1.0. Thus, the cutout area is corrected so as to approach the center of the original image data by the correction process for the original image data.

  On the other hand, when the distance between the center position of the cutout area indicated by the shake correction value (yaw, pitch) and the center coordinates of the original image data is small, the cutout zoom ratio / correction parameter determination unit 19 sets the centering coefficient α. For example, the centering coefficient α is determined to be corrected to a value close to 1.0. As a result, the cut-out area is moved away from the center of the original image data by the correction process on the original image data, but it is possible to improve the disturbance of the video due to the shake by the correction process. Further, the cut-out zoom ratio / correction parameter determination unit 19 is necessary at the time of correcting the size of the captured image data using the shake correction value (yaw, pitch) indicating the center position of the captured image data. The centering coefficient for each frame can be easily corrected.

For example, the cut-out zoom ratio / correction parameter determination unit 19 calculates {| yaw | + | pitch |} / 2 using the shake correction value (yaw, pitch). The cut-out zoom ratio / correction parameter determination unit 19
(1) When {| yaw | + | pitch |} / 2 is 25 degrees or more, it is determined to correct the centering coefficient α to “α × 1.0”.
(2) When {| yaw | + | pitch |} / 2 is 20 to 25 degrees, it is determined to correct the centering coefficient α to “α × 0.9”;
(3) When {| yaw | + | pitch |} / 2 is 15 to 20 degrees, it is determined to correct the centering coefficient α to “α × 0.8”;
(4) When {| yaw | + | pitch |} / 2 is 10 to 15 degrees, it is determined to correct the centering coefficient α to “α × 0.7”;
(5) When {| yaw | + | pitch |} / 2 is 5 to 10 degrees, it is determined to correct the centering coefficient α to “α × 0.6”.
(6) When {| yaw | + | pitch |} / 2 is 0 to 5 degrees, it is determined to correct the centering coefficient α to “α × 0.5”.

  The cut-out image generation unit 21 calculates a centering coefficient α necessary for correction processing (for example, cut-out processing) for one frame of the original image data captured by the imaging unit IM for each frame of the original image data.

  The cut-out image generation unit 21 corrects the original image data captured by the imaging unit IM based on the information on the cut-out zoom rate and the correction parameter determined by the cut-out zoom rate / correction parameter determination unit 19 (for example, The cut-out image data of the original image data is generated by performing cut-out processing and distortion correction processing. The cutout image generation unit 21 outputs the cutout image data to the video encoder 23.

For example, the cut-out image generation unit 21
(1) When the cut-out zoom rate is “60%” and the centering coefficient is “α × 1.0”, the original image data size “ 60% "is cut out, and further, the cut-out image data is generated using the centering coefficient" α × 1.0 ",
(2) When the cut-out zoom rate is “65%” and the centering coefficient is “α × 0.9”, the original image data size “ 65% "is cut out, and further, cutout image data is generated using the centering coefficient" α × 0.9 ",
(3) When the cut-out zoom rate is “70%” and the centering coefficient is “α × 0.8”, the size of the original image data is “0” with respect to the original image data captured by the imaging unit IM. 70% "is cut out, and further, cutout image data is generated using the centering coefficient" α x 0.8 "
(4) When the cut-out zoom ratio is “75%” and the centering coefficient is “α × 0.7”, the size of the original image data is “1” with respect to the original image data captured by the imaging unit IM. 75% "is cut out, and further, cutout image data is generated using the centering coefficient" α x 0.7 ".
(5) When the cut-out zoom rate is “80%” and the centering coefficient is “α × 0.6”, the original image data size “ 80% "is cut out, and cutout image data is generated using the centering coefficient" α × 0.6 "
(6) When the cut-out zoom rate is “85%” and the centering coefficient is “α × 0.5”, the size of the original image data is “1” with respect to the original image data captured by the imaging unit IM. “85%” is cut out, and cut-out image data is generated using the centering coefficient “α × 0.5”.

  Note that the cut-out image generation unit 21 generates cut-out image data by correcting the original image data based only on the information on the cut-out zoom rate determined by the cut-out zoom rate / correction parameter determination unit 19. Also good.

  The video encoder 23 uses the cut-out image data generated by the cut-out image generation unit 21 to generate encoded data for conversion into a data format in which cut-out image data can be stored and transmitted. The video encoder 23 stores the encoded data of the cut-out image data in the storage medium 25 and further outputs it to the network interface 27.

  The storage medium 25 is configured by using, for example, a hard disk device or a semiconductor memory (for example, a flash memory), and stores encoded data of cut-out image data generated by the video encoder 23. Note that the storage medium 25 is not limited to a hard disk device or a semiconductor memory built in the imaging device 100 or the image processing device 50. For example, an external connection medium (for example, a flash memory) that can be connected via a USB (Universal Serial Bus) terminal. Or a semiconductor memory).

  The network interface 27 as an example of a transmission unit uses the encoded data of the cut-out image data generated by the video encoder 23 to perform packet generation processing for transmission to the receiving device 200 that is a transmission destination, for example, and the cut-out image A packet of encoded data of data is transmitted to receiving apparatus 200. Thereby, the network interface 27 can transmit the cut-out image data obtained by the correction process on the captured image data to the external device.

  The network NW is a wireless network or a wired network. The wireless network is, for example, NFC (Near Field Communication), Bluetooth (registered trademark), IrDA, wireless LAN (Local Area Network), 3G, LTE (Long Term Evolution), or WiGig. The wired network is, for example, an intranet or the Internet.

  The receiving device 200 is an external device connected to the imaging device 100 or the image processing device 50 via the network NW, and receives encoded data of cut-out image data transmitted from the imaging device 100 or the image processing device 50. The receiving device 200 decodes the encoded data of the clipped image data and displays the clipped image data on, for example, a display device (not shown).

  Although not shown in FIG. 2, the imaging apparatus 100 may be used alone when the imaging apparatus 100 includes a battery unit including a power source and a secondary battery. The housing of the imaging device 100 and the housing in which the battery unit BT is incorporated may be connected via a cable (see FIG. 3A). FIG. 3A is an external perspective view of the imaging device 100 of the present embodiment connected to the battery unit BT by a cable.

  In addition, although the imaging device 100 is used in a state where it is mainly worn by a person, the usage pattern of the imaging device 100 is not limited to the usage pattern shown in FIG. FIG. 3B is a diagram illustrating an example of a state where the imaging apparatus 100 according to the present embodiment is worn by a person.

(Operation procedure of imaging device)
Next, an operation procedure of the imaging apparatus 100 according to the present embodiment will be described with reference to FIGS. 5 and 6A to 6F. FIG. 5 is a flowchart illustrating an example of a detailed operation procedure of the imaging apparatus 100 according to the present embodiment. FIGS. 6A to 6F are diagrams illustrating an example of the cutout areas RN1 to RN6 of the cutout image data corresponding to the shake correction value. The operation procedure shown in FIG. 5 is processing for one frame of original image data in the imaging apparatus 100.

  In FIG. 5, the imaging device 100 is configured to extract original image data captured by the imaging unit IM based on the vibration information of the imaging device 100 stored in the vibration information storage unit 17 in the cut-out zoom ratio / correction parameter determination unit 19. A shake correction value (yaw, pitch) indicating the center position of the cutout area of one frame is calculated (S11), and {| yaw | + | pitch |} / 2 is further calculated using the shake correction value (yaw, pitch). Is calculated.

  When {| yaw | + | pitch |} / 2 is 25 degrees or more (S12, YES), the imaging apparatus 100 indicates the shaking state of one frame of the original image data imaged by the imaging unit IM. The vibration level is determined to be 5 (S13), and the centering coefficient α is determined to be corrected to “α × 1.0”. When the vibration level indicating the shaking state of one frame of the original image data is 5, the imaging apparatus 100 has a size of one frame of the original image data captured by the imaging unit IM and a size of one frame of the cut-out image data. The ratio (cutout zoom rate) is determined to be 60%.

  The imaging apparatus 100 uses the image data captured by the imaging unit IM based on information indicating that the cut-out zoom rate is “60%” and information indicating that the centering coefficient is corrected to “α × 1.0”. On the other hand, “60%” of the size of the original image data is cut out, and further, cut-out image data is generated using the centering coefficient “α × 1.0” (S14, see FIG. 6F). The cutout image data generated in step S14 has a size corresponding to the cutout region RN6 shown in FIG. 6F with respect to the size of the original image data.

  When {| yaw | + | pitch |} / 2 is not 25 degrees or more (S12, NO) and 20 to 25 degrees (S15, YES), the imaging apparatus 100 captures an image with the imaging unit IM. The vibration level indicating the shaking state of one frame of the original image data is determined as 4 (S16), and further, it is determined that the centering coefficient α is corrected to “α × 0.9”. When the vibration level indicating the shaking state of one frame of the original image data is 4, the imaging apparatus 100 has a size of one frame of the original image data captured by the imaging unit IM and a size of one frame of the cut-out image data. Is determined to be 65%.

  The image capturing apparatus 100 uses the image data captured by the image capturing unit IM based on information indicating that the cut-out zoom rate is “65%” and information indicating that the centering coefficient is corrected to “α × 0.9”. On the other hand, “65%” of the size of the original image data is cut out, and further, cut-out image data is generated using the centering coefficient “α × 0.9” (S17, see FIG. 6E). The cutout image data generated in step S17 has a size corresponding to the cutout region RN5 shown in FIG. 6E with respect to the size of the original image data.

  In the imaging apparatus 100, when {| yaw | + | pitch |} / 2 is not 20 to 25 degrees or more (S15, NO) and 15 to 20 degrees (S18, YES), the imaging unit IM Is determined to be 3 (S19), and the centering coefficient α is determined to be corrected to “α × 0.8”. When the vibration level indicating the shaking state of one frame of the original image data is 3, the imaging apparatus 100 has the size of one frame of the original image data captured by the imaging unit IM and the size of one frame of the cut-out image data. Is determined to be 70%.

  The imaging apparatus 100 uses the information indicating that the cut-out zoom rate is “70%” and the information indicating that the centering coefficient is corrected to “α × 0.8” as the original image data captured by the imaging unit IM. On the other hand, “70%” of the size of the original image data is cut out, and further, cut-out image data is generated using the centering coefficient “α × 0.8” (S20, see FIG. 6D). The cutout image data generated in step S20 has a size corresponding to the cutout region RN4 illustrated in FIG. 6D with respect to the size of the original image data.

  In the imaging apparatus 100, when {| yaw | + | pitch |} / 2 is not 15 to 20 degrees or more (S15, NO) and is 10 to 15 degrees (S21, YES), the imaging unit IM Is determined to be 2 (S22), and the centering coefficient α is determined to be corrected to “α × 0.7”. When the vibration level indicating the shaking state of one frame of the original image data is 2, the imaging apparatus 100 has the size of one frame of the original image data captured by the imaging unit IM and the size of one frame of the cut-out image data. And the ratio (cutting zoom rate) is determined to be 75%.

  The imaging apparatus 100 uses the information indicating that the cut-out zoom rate is “75%” and the information indicating that the centering coefficient is corrected to “α × 0.7” as the original image data captured by the imaging unit IM. On the other hand, “75%” of the size of the original image data is cut out, and further, cut-out image data is generated using the centering coefficient “α × 0.7” (S23, see FIG. 6C). The cutout image data generated in step S23 has a size corresponding to the cutout region RN3 shown in FIG. 6C with respect to the size of the original image data.

  In the imaging apparatus 100, when {| yaw | + | pitch |} / 2 is not 10-15 degrees or more (S21, NO) and 5-10 degrees (S24, YES), the imaging unit IM Is determined to be 1 (S25), and the centering coefficient α is determined to be corrected to “α × 0.6”. When the vibration level indicating the shaking state of one frame of the original image data is 1, the imaging apparatus 100 has the size of one frame of the original image data captured by the imaging unit IM and the size of one frame of the cut-out image data. Is determined to be 80%.

  The imaging apparatus 100 uses the information indicating that the cut-out zoom rate is “80%” and the information indicating that the centering coefficient is corrected to “α × 0.6” as the original image data captured by the imaging unit IM. On the other hand, “80%” of the size of the original image data is cut out, and further, cut-out image data is generated using the centering coefficient “α × 0.6” (S26, see FIG. 6B). The cutout image data generated in step S26 has a size corresponding to the cutout area RN2 shown in FIG. 6B with respect to the size of the original image data.

  When {| yaw | + | pitch |} / 2 is not 5 to 10 degrees or more (S24, NO), the imaging apparatus 100 shakes one frame of the original image data captured by the imaging unit IM. Is determined to be 0 (S27), and the centering coefficient α is determined to be corrected to “α × 0.5”. When the vibration level indicating the shaking state of one frame of the original image data is 0, the imaging device 100 has a size of one frame of the original image data captured by the imaging unit IM and a size of one frame of the cut-out image data. Is determined to be 85%.

  The imaging apparatus 100 uses the information indicating that the cut-out zoom rate is “85%” and the information indicating that the centering coefficient is corrected to “α × 0.5” as the original image data captured by the imaging unit IM. On the other hand, “85%” of the size of the original image data is cut out, and further, cut-out image data is generated using the centering coefficient “α × 0.5” (S28, see FIG. 6A). The cutout image data generated in step S28 has a size corresponding to the cutout region RN1 shown in FIG. 6A with respect to the size of the original image data.

  As described above, the imaging apparatus 100 according to the present embodiment corrects the original image data captured by the imaging unit IM based on the vibration information detected by the vibration sensor 15 according to the movement of the user of the imaging apparatus 100. Each piece of information on the cutout zoom ratio and the correction parameter (for example, the centering coefficient α) that is sometimes required is determined, and cutout image data of the original image data is generated using the determined pieces of information.

  As a result, the imaging device 100 reduces the size of the cutout area during correction processing (for example, cutout processing) of the captured image data when a large shake occurs according to the movement of the user. Even in the case of occurrence of cutout, cut-out image data in which the disturbance of the image due to shaking is suppressed can be obtained. In addition, the imaging apparatus 100 increases the size of the cut-out area when the captured image data is corrected when a small fluctuation occurs according to the user's movement. Cut-out image data with high image quality can be obtained. In other words, the imaging apparatus 100 can provide the optimum video quality while suppressing the viewer's uncomfortable feeling by performing adaptive image correction according to the magnitude of the user's shake.

  While various embodiments have been described above with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  Note that the imaging apparatus 100 or the image processing apparatus 50 according to the present embodiment generates a plurality of original images when generating cutout image data having a predetermined size (60%) from the size (100%) of the original image data, for example. The size of the cut-out area may be changed for each frame of the original image data so that the cut-out image data has a size of 60% over the data frame.

  As a result, the image capturing apparatus 100 or the image processing apparatus 50 performs, for example, the size of the cut-out image data with respect to the image data obtained by image capturing during the image data correction processing using the vibration information or the shake correction value (yaw, pitch). The ratio can be changed from 100% to decrease by 5% for each frame.

  Therefore, the imaging device 100 or the image processing device 50 has a sharp angle of view of the cut-out image data, for example, the ratio of the size of the cut-out image data to the original image data obtained by the image pickup is changed from 100% to 60%. It is possible to avoid a sense of discomfort in the eyes that accompanies changes.

  The present invention provides an image processing apparatus, an imaging apparatus, and an image that can provide an optimal video quality by suppressing an uncomfortable feeling of a viewer by performing adaptive image correction according to the magnitude of a user's shake. It is useful as a processing method.

11 Image sensor 13 Image data storage unit 15 Vibration sensor 17 Vibration information storage unit 19 Cutout zoom ratio / correction parameter determination unit 21 Cutout image generation unit 23 Video encoder 25 Storage media 27 Network interface 50 Image processing device 100 Imaging device 200 Reception device LN Lens NW network

Claims (7)

An image data storage unit for storing captured image data;
A vibration information storage unit for storing vibration information at the time of imaging of the image data;
A cutout region determination unit that determines the size of the cutout region of the captured image data in accordance with the vibration information stored in the vibration information storage unit;
A cutout image generation unit that generates cutout image data of the image data stored in the image data storage unit according to the size of the cutout region determined by the cutout region determination unit;
Image processing device. The image processing apparatus according to claim 1,
The cutout area determination unit
In accordance with the vibration information stored in the vibration information storage unit, a shake correction value indicating the center of the cutout region is calculated,
According to the calculated shake correction value, a ratio of the size of the cut-out image data to the size of the captured image data is determined.
Image processing device. The image processing apparatus according to claim 1,
The cutout area determination unit
In accordance with the vibration information stored in the vibration information storage unit, a shake correction value indicating the center of the cutout region is calculated,
According to the calculated shake correction value, the centering coefficient for each frame of the captured image data is corrected.
Image processing device. The image processing apparatus according to claim 1, wherein:
The cutout area determination unit
Changing the size of the cutout region for each frame so that the size of the predetermined cutout region is over a plurality of frames of the image data;
Image processing device. An image processing apparatus according to any one of claims 1 to 4, wherein
A transmission unit that transmits the cut-out image data generated by the cut-out image generation unit to an external device connected via a network;
Image processing device. An imaging unit for imaging video;
A vibration sensor for detecting vibration information at the time of imaging the image;
A cutout area determination unit that determines the size of the cutout area of the image data of the video imaged by the imaging unit in accordance with the vibration information detected by the vibration sensor;
A cutout image generation unit that generates cutout image data of the image data captured by the imaging unit according to the size of the cutout region determined by the cutout region determination unit;
Imaging device. An image processing method in an image processing apparatus having captured image data and vibration information at the time of imaging of the image data,
Determining a size of a cut-out area of the imaged image data according to the vibration information;
Generating cutout image data of the image data in accordance with the determined size of the cutout region,
Image processing method.

Images