JP2006295626A - Fish-eye image processing apparatus, method thereof and fish-eye imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently correct the blurring quantity of a fish-eye image. <P>SOLUTION: A fish-eye image processing apparatus for correcting the distortions of a fish-eye image picked up by a fish-eye optical system detects the blurring quantity of the fish-eye image (S202), calculates the blurring quantity of a peripheral part of the fish-eye image, on the basis of the detected blurring quantity and determines the segmentation area of the fish-eye image, on the basis of the calculation result (S203). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an apparatus and method for correcting distortion of a fisheye imaged using a fisheye optical system, and an imaging apparatus.

  Conventionally, there is a method of electrically correcting image blur due to camera shake in a video camera. In this method, a region to be actually recorded or output is selected as a cutout region from images formed on the image pickup surface, and an image pickup device having a pixel number larger than the number of pixels necessary for image pickup is used. Is corrected by moving the image according to the amount of camera shake. For this reason, there is a problem that the imaging surface cannot be effectively used, resolution is deteriorated, and the angle of view of the output image is narrower than the optical characteristics.

On the other hand, for example, in Patent Document 1, the size of an area (margin area) in which the position of an image to be extracted can be moved is determined based on the zoom position, and the extraction position on the imaging surface is determined based on the amount of camera shake. Has been decided.
JP-A-6-245134 (paragraph numbers 0015-0019, FIG. 1, FIG. 5 and FIG. 6)

  However, Patent Document 1 corrects blurring in a perspective projection image in which blurring in the image is uniform, and the apparent blurring amount that appears in the moving image by distorting the subject shape of the photographed image is the periphery of the image. It is not possible to efficiently correct the blur of a fish-eye image that has a characteristic of becoming smaller as it is compressed at the part.

  The present invention has been made in view of the above-described problems, and an object thereof is to efficiently correct the blur amount of a fisheye image.

  A first aspect of the present invention relates to a fish-eye image processing apparatus that corrects distortion of a fish-eye image captured using a fish-eye optical system, and a shake analysis unit that detects a shake amount of a fish-eye image; A peripheral blur amount calculation unit that calculates the blur amount of the peripheral part of the fisheye image based on the blur amount detected by the blur analysis unit, and the fisheye image based on the calculation result of the peripheral blur amount calculation unit And a cutout region determination unit that determines the cutout region.

  A second aspect of the present invention relates to a fish-eye image capturing apparatus, and includes an image capturing unit that captures a fish-eye image using a fish-eye optical system and the fish-eye image processing apparatus. .

  A third aspect of the present invention relates to a fish-eye image processing method for correcting distortion of a fish-eye image captured using a fish-eye optical system, and a detection step for detecting a blur amount of the fish-eye image, and the detection A calculation step of calculating a blur amount of a peripheral portion of the fish-eye image based on the blur amount detected in the step; a determination step of determining a cut-out region of the fish-eye image based on a calculation result in the calculation step; , Including.

  According to the present invention, it is possible to efficiently correct the blur amount of a fisheye image.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the embodiments described below.
(First embodiment)
FIG. 1 is a diagram showing a configuration of a fish-eye image capturing apparatus 100 according to a preferred first embodiment of the present invention. The fish-eye image capturing apparatus 100 includes a fish-eye optical system 101 that projects a fish-eye image, an image sensor 102 as an image capturing unit, and a fish-eye image processing apparatus. The fish-eye image processing apparatus includes, for example, an image forming circuit 103, field memories 107 and 108, a memory control circuit 109, a shake analysis unit 110, a projection conversion information calculation unit 113, a projection conversion unit 115, an image output unit 116, and a clipping unit 118. Etc.

  Reference numeral 103 denotes an image forming circuit that performs image processing on an electrical signal output from the image sensor 102 to form a video information signal. The image forming circuit 103 includes an A / D conversion circuit 104 that converts an analog signal into a digital signal, an auto gain control circuit (AGC) 105 that performs digital signal level correction, and an auto white balance circuit (AWB) that performs video white level correction. ) And the like, and form a digital fisheye image.

  Reference numerals 107 and 108 denote field memories for temporarily storing and holding captured video signals in units of one screen or a plurality of screens. Reference numeral 107 denotes field memories 1 and 108 for storing and holding image-formed fisheye images. This is a field memory 2 that temporarily stores and holds an image after fisheye image distortion correction processing, which will be described later. Reference numeral 109 denotes a memory control circuit that controls video signals input to and output from the field memories 107 and 108. Reference numeral 110 denotes a shake analysis unit that analyzes the tendency of motion detection and motion between adjacent video fields, and includes a shake amount detection unit 111 and a shake amount analysis unit 112. The shake amount detection unit 111 detects the shake amount of the fisheye image captured using the fisheye optical system 101. The shake amount analysis unit 112 calculates the shake amount of the entire fisheye image based on the shake amount detected by the shake amount detection unit 111.

  113 is a projection conversion information calculation unit that generates a conversion formula for performing projection conversion between a fisheye image and a perspective projection image by the CPU, and 114 is a CPU that calculates a cutout region for image stabilization in the fisheye image. A cut-out area determining unit 115 for projectively converting the cut-out area in the fish-eye image into a perspective projection image to generate a camera shake correction image; and 116, a video not shown in FIG. An image output unit 117 for displaying or recording on the output device is a peripheral blur amount calculating unit 117 that calculates the blur amount of the peripheral portion of the fisheye image based on the output from the blur analysis unit 110. The cutout unit 118 includes a cutout region determination unit 114 and a peripheral blur amount calculation unit 117. As a method for calculating the amount of blur at the periphery of the fisheye image, for example, the amount of blur at the periphery of the fisheye image is calculated using a predetermined conversion formula for the amount of blur detected at a predetermined portion (for example, the center) of the fisheye image. It is possible to use a method of calculating by converting the fisheye image into a perspective projection image and calculating the shake amount of the entire fisheye image, and then backprojecting the calculated shake amount of the entire fisheye image. Then, a method of calculating the amount of blur around the fisheye image may be used. In addition, various methods can be used as a method of calculating the amount of blur around the fisheye image.

  The operation of this apparatus configured as described above will be described below. An example of the camera shake correction method and operation in the present embodiment will be described with reference to the flowchart shown in FIG.

  In step S <b> 201, a subject image projected by the fisheye optical system 101 of FIG. 1 is formed on the image sensor 102. In the fish-eye optical system 101, light from an incident angle of 0 to 90 degrees can be imaged on the image sensor 102. However, the projection image is distorted by, for example, an equidistant projection method in which the distance from the optical axis is proportional to the incident angle or an orthogonal projection method in which the distance is proportional to the sine function. A subject video image formed on the image sensor 102 is photoelectrically converted and input to the image forming circuit 103 as an analog video signal corresponding to the subject luminance. In the image forming circuit 103, the analog signal is converted into, for example, a 14-bit digital signal by the A / D conversion circuit 104. The digital signal is further subjected to signal level correction and white level correction by an auto gain control circuit (AGC) 105 and an auto white balance circuit (AWB) 106, and is temporarily recorded and held in the field memory 107.

  FIGS. 3A and 3B are diagrams schematically illustrating a subject and a subject video by the fish-eye optical system 101. FIG. 301 in FIG. 3A is a diagram showing a grid-like subject, and 302 in FIG. 3B is a projection of the subject 301 in FIG. 3A onto the image sensor 102 by the fisheye optical system 101 in an orthogonal projection. It is a figure which shows the fish-eye image when it does. The fisheye image 302 that has been subjected to the above-described image forming process is stored and held in the field memory 107 of FIG.

  In this apparatus, field images are sequentially captured at a determined frame rate, and subject images stored and held in the field memory 107 are input to the blur analysis unit 110 together with continuous field images. At the same time, the field memory 107 is rewritten and updated with the current field image. The above operation is controlled by the memory control circuit 109.

  In step S202, the blur amount detection unit 111 detects a corresponding region as a motion vector by a template matching method for a plurality of continuous field video regions. The motion vector detected in this way becomes the apparent “blurring amount” of each region appearing in the image. Here, the motion vector detection method is not limited to the template matching method, and another image processing method such as a gradient method or a motion detection device such as a gyro may be used.

  The motion vectors detected in a plurality of regions by the blur amount detection unit 111 are integrated by the blur amount analysis unit 112 to generate a motion vector representing the blur of the entire image. In the fish-eye image, the image around the image becomes smaller as it is compressed, so even the motion vector representing the same motion has a different apparent size depending on the position in the fish-eye image. For this reason, it is extremely difficult to integrate motion vectors of a plurality of regions while maintaining a fish-eye image. Therefore, in order to integrate motion vectors, motion vectors of a plurality of regions obtained on a fisheye image are converted into motion vectors in a perspective projection image without distortion, and then motion vectors are integrated by a shake amount analysis unit 112. It is desirable to output the motion vector in the perspective projection image.

  In step S203, a projection conversion formula from a fisheye image to a perspective projection image is generated. The projection conversion information calculation unit 113 is a CPU for calculating a projection conversion formula for converting a fisheye image into a perspective projection image that is an image captured by a normal lens. Here, a case where an orthographic fish-eye image is converted into a perspective projection image will be described as an example. The relationship between the image height y and the incident angle θ in the orthographic fisheye image is as follows.

…（数式１）
と表される。また、透視投影像は像高をＹとすると

... (Formula 1)
It is expressed. Further, the perspective projection image has an image height Y.

…（数式２）
と表される。ここで、Ｆは透視投影像の焦点距離であり、出力画像の仮想的な焦点距離として任意に定めることができる。このとき、変換の前後において入射角の値は変化しないので、

... (Formula 2)
It is expressed. Here, F is a focal length of the perspective projection image, and can be arbitrarily determined as a virtual focal length of the output image. At this time, since the value of the incident angle does not change before and after the conversion,

…（数式３）
より、

... (Formula 3)
Than,

…（数式４）
となる。数式４が射影変換式であり、周辺部ぶれ量算出部１１７へと送られる。ここでは魚眼像の射影方式として正射影を用いて説明したが、等距離射影ｙ＝ｆθ等の他の射影方式を用いてもよい。

... (Formula 4)
It becomes. Formula 4 is a projective transformation formula, which is sent to the peripheral blur amount calculation unit 117. Although the description has been made here using orthographic projection as a fisheye image projection method, other projection methods such as equidistant projection y = fθ may be used.

  In addition, in order to projectively convert a motion vector on an image to a perspective projection image using the above projective transformation formula, by performing projective transformation on the start point and end point of the motion vector, the motion vector after the projective transformation is obtained. You can get the start and end points.

  In this embodiment, the projection conversion information calculation unit 113 is a CPU, and the calculated projection conversion formula is sent to the blur amount analysis unit 112. However, each pixel before and after the projection conversion is used by using an LUT instead of the CPU. The correspondence between the addresses may be stored and held, and necessary pixel addresses may be sent to the blur amount analysis unit 112 as appropriate.

  Here, a difference between apparent motion vectors appearing on the fisheye image and the perspective projection image will be described. FIG. 4 is a diagram showing an apparent motion vector in a perspective projection image, where 401 is an imaging region, 402 is a cutout region for image stabilization, and 403 is an image center representing the blurring of the entire image obtained by integrating the motion vectors of each region. An apparent motion vector (hereinafter referred to as an “image overall motion vector”) in the area, and 404 an apparent image overall motion vector (hereinafter referred to as an “image periphery motion vector”) in the periphery of the image. In the perspective projection image, the apparent motion vector in the imaging region 401 is uniform in the entire region. Therefore, the image center motion vector 403 and the image peripheral motion vector 404 are vectors having the same size and direction. Therefore, if the cutout area 402 is determined so that the image center motion vector 403 is included in the imaging area 401 without excess or deficiency, it is possible to perform camera shake correction without causing wasted pixels or missing images. Become.

  On the other hand, FIG. 5 is a diagram showing an apparent motion vector in a fish-eye image, in which 501 is an imaging region, 502 is a clipping region for image stabilization, and 503 is an apparent motion vector that represents blurring of the entire image at the center of the image. , 504 represent apparent overall image motion vectors at the periphery of the image. In a fish-eye image, the image peripheral motion vector 504 is smaller than the motion vector 503 at the center of the image because the image is distorted as if the image was compressed at the periphery of the imaging region 501. Accordingly, as in the case of the perspective projection image, when the cutout region is determined so that the motion vector 503 in the center of the image is included in the imaging region 501, a margin having a size larger than the apparent motion vector is provided in the peripheral portion of the image. As a result, useless pixels are generated. In this embodiment, in order to perform efficient image stabilization, the cutout region 502 is determined so that the image peripheral motion vector 504 is included in the imaging region 501 without excess or deficiency as shown in FIG. As described above, by setting a cut-out area that does not cause waste in consideration of the characteristics of the fisheye image, it is possible to perform an efficient camera shake correction that suppresses the narrowing of the output image and the deterioration of the image quality.

  In FIG. 5, the cutout region 502 for image stabilization has a barrel shape because the shape of the cutout region 502 after the projective transformation when the projective transformation from the fisheye image to the perspective projection image is performed thereafter. This is because the backward calculation is performed so as to form a rectangle. As a result, pixel loss before and after projective transformation can be prevented, and the cutout area in the fisheye image can be made as large as possible.

  In step S204, based on the projective transformation formula calculated in step S203, a motion vector in the perspective projection image is calculated by the blur amount analysis unit 112, and based on this motion vector, the peripheral portion blur amount calculation unit 117 calculates the fisheye. The amount of blurring at the periphery of the image is calculated. A motion vector representing the blur of the entire image in the perspective projection image is input from the blur analysis unit 110 to the peripheral blur amount calculation unit 117. Since the cut-out area for image stabilization in the fisheye image is determined by the apparent size of the image peripheral motion vector, the image of the perspective projection image is calculated using the projection conversion formula calculated by the projection conversion information calculation unit 113. The entire motion vector is converted into a motion vector around the image of the fisheye image. Then, the range of the cutout region is determined by the cutout region determination unit 114 so that the converted image peripheral motion vector is included in the imaging region without excess or deficiency. Thereafter, the cutout region is shifted in the imaging surface so as to cancel the image peripheral motion vector representing the blur of the entire image in the fisheye image. With the above operation, it is possible to determine a cutout region for camera shake correction. Each pixel in the determined cutout region is associated with the coordinate value from the fisheye image to the perspective projection image using the projective transformation formula, and is sent to the projective transformation unit 115 as conversion address data.

  In step S <b> 205, the projection conversion unit 115 is stored and held in the field memory 1 107 using the address conversion information to the perspective projection image of the cut-out area of the fisheye image sent from the cut-out area calculation unit 114. Pixel values are read out from a fisheye image. 6A to 6C illustrate the function of the projective transformation unit 115. FIG. Reference numeral 601 in FIG. 6A denotes a fisheye image stored and held in the field memory 107. The distortion of the fish-eye image is determined by the projection method of the fish-eye zoom optical system. For example, if the incident angle of light rays from the subject to the fish-eye zoom optical system 101 is θ, the image height y from the optical axis is y = fθ. There are an equidistant projection represented, an orthographic projection represented by y = fsinθ, and the like. In the present embodiment, a fisheye image projected by the orthogonal projection method is represented by 601 as an example, but the present invention is not limited to this. The projective transformation unit 115 performs projective transformation of the fisheye image cutout area 6021 stored and held in the field memory 107 as in 602 in FIG. 6B to a perspective projection image represented by y = ftanθ. Output to the field memory 108. At this time, if camera shake exists in the fish-eye image input from the field memory 107, projective transformation is performed after shifting the cutout area in the direction to cancel the camera shake like the cutout area 6022. Reference numeral 603 in FIG. 6C represents a memory space of the field memory 108 and a state in which the perspective projection image after the projective transformation of the projection transformation target area 6021 is developed in the memory space 603. The lattice-like subject image represented by 301 in FIG. 3A is distorted as shown by 601 in FIG. 6A by orthographic projection, and the distortion is corrected by the projective transformation unit. . When the image after projective transformation is smaller than the size of the output image, the projective transformation unit 115 appropriately performs enlargement / reduction processing so as to match the output image size. Through the above processing, the perspective projection image subjected to the camera shake correction is accumulated in the field memory 2 108.

  In the flowchart shown in FIG. 2, as a method for calculating the blur amount of the periphery of the fisheye image, the fisheye image is converted into a perspective projection image and the blur amount of the entire fisheye image is calculated, and then the entire fisheye image is calculated. Although the method of calculating the amount of blurring of the peripheral portion of the fisheye image by performing back projection conversion of the amount of blurring is shown as an example, the present invention is not limited to this, and other methods may be used.

  As described above, according to the present embodiment, by using the characteristic that the apparent motion vector in the peripheral portion of the fisheye image is smaller than the motion vector in the perspective projection image, a larger cutout region can be obtained. By setting, it is possible to reduce the angle of view and image quality deterioration of the image stabilization image.

(Second Embodiment)
The second embodiment is an embodiment where the fish-eye image capturing apparatus 100 according to the first embodiment has a zoom function. FIG. 7 is a diagram showing a configuration of a fish-eye image capturing apparatus 700 according to the preferred second embodiment of the present invention. In addition to FIG. 1, a zoom control unit 701 for controlling the zoom position (focal length) of the fisheye optical system and a zoom position acquisition unit 702 for detecting the zoom position are added.

  Hereinafter, an example of the camera shake correction method and operation in the present embodiment will be described with reference to the flowchart shown in FIG.

  Steps S801 and S802 are performed in the same manner as steps S201 and S202 of FIG.

  In step S803, the zoom position of the fish-eye zoom optical system 101 is controlled by the zoom control unit 701, and the zoom position of the fish-eye zoom optical system is detected as the focal distance by the zoom position acquisition unit 702. Here, the zoom position information may be detected as information such as a magnification in addition to the focal length.

  In step S804, a projection conversion formula from a fisheye image to a perspective projection image is generated. The projection conversion information calculation unit 113 is a CPU for calculating a projection conversion formula for converting a fisheye image into a perspective projection image that is an image captured by a normal lens. This conversion formula is a fisheye zoom optical system. Since the zoom position changes depending on the zoom position 101, the zoom position acquisition unit 701 is regenerated every time a change in the zoom position is detected.

  Here, the relationship between the zoom position of the fisheye optical system and the apparent motion vector on the screen will be described. FIG. 9 is a diagram showing how the apparent magnitude of the motion vector representing the blur of the entire image changes in the center of the image when the zoom position of the fish-eye zoom optical system is changed. The zoom position is taken, and the vertical axis is the magnitude of the motion vector. Here, the motion vector representing the blur of the entire image in the perspective projection image calculated by the blur analysis unit 110 is converted into a motion vector in the center of the image of the fisheye image, and how the magnitude changes depending on the zoom position. It is calculated whether to do. A projection conversion formula from the motion vector of the perspective projection image to the motion vector of the fisheye image can be obtained as follows. Solving Formula 2 of the perspective projection image with respect to the incident angle θ,

…（数式５）
となり、数式５と正射影の数式１により、

... (Formula 5)
Then, using Formula 5 and Orthographic Formula 1,

…（数式６）
となる。数式６が透視投影像の動きベクトルから魚眼像の動きベクトルへの射影変換式となる。

... (Formula 6)
It becomes. Formula 6 is a projection conversion formula from the motion vector of the perspective projection image to the motion vector of the fisheye image.

  Here, orthographic projection is used as the fish-eye image projection method, but other projection methods such as equidistant projection y = fθ may be used. FIG. 9 shows that the magnitude of the motion vector changes linearly with respect to the zoom position at the center of the image.

  On the other hand, FIGS. 10A and 10B are diagrams showing the relationship between the image center of the fisheye image and the ratio of motion vectors at each angle of view position and the zoom position. FIG. 10A is a diagram showing how the ratio between the magnitude of the motion vector and the magnitude of the motion vector at the center of the image changes when the zoom position of the fisheye optical system changes. Is the zoom position, and the vertical axis is the ratio. FIG. 10B is a diagram illustrating the angle-of-view positions A to C in the fish-eye image of FIG.

  The ratio between the magnitude of the motion vector representing the blur of the whole image in the perspective projection image and the apparent magnitude of the whole image motion vector at each angle of view is calculated as follows.

  First, if the image heights of the start and end points of the motion vector ν representing the blur of the entire image in the perspective projection image are νs and νe, respectively, the magnitude is νe−νs. At this time, the position of νs and νe on the image is set to each angle of view position. Then, νs and νe are converted into a fish-eye image height using Equation 6. Assuming that the converted motion vector is ν ′ and the image heights of the start and end points are νs ′ and νe ′, respectively, the apparent magnitude of the entire image motion vector at each angle of view of the fisheye image is νe′−νs ′. It can be expressed as. Therefore, the ratio of the magnitude of the motion vector representing the blur of the whole image in the perspective projection image and the apparent magnitude of the whole image motion vector at each angle of view position is

…（数式７）
となる。

... (Formula 7)
It becomes.

  By changing the focal length f of the fish-eye image in Equation 6 and solving Equation 7, the relationship between the zoom position and the ratio can be calculated.

  From FIG. 10A, the apparent size of the entire image motion vector in the image peripheral portion A becomes smaller than the original entire image motion vector as the zoom position is on the wide angle side (WIDE), and the telephoto side (TELE). It can be seen that the closer to is the same size as the original whole image motion vector.

  The apparent size of the whole image motion vector in the image intermediate portion B is also smaller than the original whole image motion vector as the zoom position is at the wide angle side (WIDE), and the original image as a whole is closer to the telephoto side (TELE). It can be seen that it approaches the same magnitude as the motion vector. Compared with the image peripheral part A, the image intermediate part B is smaller than the original whole image motion vector as a whole, and the tendency is more prominent as it is on the wide angle side (WIDE).

  Similarly, the apparent size of the entire image motion vector in the image center C is smaller than the original entire image motion vector as the zoom position is on the wide-angle side (WIDE), and the original size as the telephoto side (TELE) is located. It can be seen that it approaches the same magnitude as the entire image motion vector. The image intermediate portion C is further smaller than the original image whole motion vector as compared with the image peripheral portion B, and the tendency is more prominent as it is on the wide angle side (WIDE).

  As described above, the apparent magnitude of the motion vector changes nonlinearly with respect to the zoom position at any of the view angle positions A to C. Therefore, in the camera shake correction of the fisheye image, if the range of the cutout region is linearly changed according to the size of the entire image motion vector calculated by the shake analysis unit 110 and the zoom position of the fisheye optical system, On the wide-angle side, a margin larger than the apparent size of the entire image motion vector in the peripheral portion of the image is secured, and pixels are wasted in camera shake correction.

    Therefore, the entire image motion vector in the perspective projection image is converted into a motion vector around the image of the fisheye image, and the range of the cutout region is shown in FIG. 10 according to the size of the image periphery motion vector and the zoom position of the fisheye optical system. As shown in the figure, camera shake correction that does not waste pixels can be realized by changing non-linearly.

  In the flowchart shown in FIG. 8, the case where the zoom position detection in step S803 is performed after steps S801 and S802 is shown as an example. However, the present invention is not limited to this, and step S803 includes steps S801 and S802. You may go before.

  As described above, according to the present embodiment, when the zoom position of the fisheye zoom optical system is changed, it is considered that the apparent magnitude of the motion vector in the peripheral portion of the image changes nonlinearly. In accordance with this, the size of the cut-out area is also changed nonlinearly, so that it is possible to perform efficient camera shake correction without causing missing pixels or images that are wasted during camera shake correction.

(Third embodiment)
The third embodiment forms an image projected by the fish-eye zoom optical system 101 using the image sensor 102 in the first embodiment, whereas a captured fish-eye moving image is a moving image. It is one Embodiment in the case of acquiring from a recording part. FIG. 11 is a diagram showing a configuration of a fish-eye image capturing apparatus 1100 according to a preferred third embodiment of the present invention. In place of the fish-eye optical system 101, the image sensor 102, and the image forming circuit 103 in FIG. This is a fish-eye image stabilization reproduction device including a moving image recording unit 1101.

  The moving image recording unit 1101 records and saves the captured moving image, and reads it out as necessary. The fish-eye moving image recorded in the moving image recording unit 1101 is read for each field and sequentially stored in the field memory 107. Subsequent processing is the same as that in the first embodiment, and it is possible to perform camera shake correction as post-processing for a captured fish-eye moving image.

  As described above, according to the present embodiment, camera shake correction can be performed on a captured fish-eye moving image by using the moving image reproducing unit.

(Fourth embodiment)
The fourth embodiment is an embodiment in the case where a zoom function is provided to the fish-eye image stabilization reproduction device according to the third embodiment. FIG. 12 is a diagram showing a configuration of a fish-eye image capturing apparatus 1200 according to the preferred fourth embodiment of the present invention. In addition to FIG. 11, a zoom control unit 1201 for controlling the zoom position of the fisheye optical system and a zoom position acquisition unit 1202 for detecting the zoom position are added.

  The fish-eye moving image recorded in the moving image recording unit 1101 is read for each field and sequentially stored in the field memory 107. At this time, zoom position information is added to each field of the moving image together with the image data. This zoom position information is read by the zoom position acquisition unit 1202 and sent to the projective conversion information calculation unit 113. Subsequent processing is the same as in the second embodiment, and it is possible to perform camera shake correction in accordance with the zoom position as post-processing for a captured fish-eye moving image.

  As described above, according to this embodiment, it is possible to perform an efficient camera shake correction according to the zoom position even for a captured fish-eye moving image.

It is a figure which shows the structure of the fish-eye image imaging device which concerns on the suitable 1st Embodiment of this invention. It is a flowchart of the camera-shake correction process which concerns on suitable 1st Embodiment of this invention. It is a figure which shows the fish-eye image projected on the image pick-up element by the grid | lattice object and the fish-eye optical system by the orthogonal projection. It is a figure which shows the amount of apparent blurring in a perspective projection image. It is a figure which shows the amount of apparent blurring in a fisheye image. It is a figure which shows typically the functional effect of a projective transformation part. It is a figure which shows the structure of the fish-eye image imaging device which concerns on the suitable 2nd Embodiment of this invention. 10 is a flowchart of camera shake correction processing according to a preferred second embodiment of the present invention. It is a figure showing the relationship between the image center motion vector and zoom position in a perspective projection image. It is a figure which shows the relationship between each image position in a fisheye image, the ratio of the motion vector of a peripheral part, and a zoom position. It is a figure which shows the structure of the fish-eye image imaging device which concerns on the suitable 3rd Embodiment of this invention. It is a figure which shows the structure of the fish-eye image imaging device which concerns on the suitable 4th Embodiment of this invention.

Claims (8)

A fish-eye image processing apparatus that corrects distortion of a fish-eye image captured using a fish-eye optical system,
A shake analysis unit that detects the shake amount of the fisheye image;
A peripheral blur amount calculating unit that calculates a blur amount of the peripheral part of the fish-eye image based on the blur amount detected by the blur analysis unit;
A cutout region determination unit that determines a cutout region of the fisheye image based on the calculation result of the peripheral blur amount calculation unit;
A fish-eye image processing apparatus comprising:   The fisheye image processing apparatus according to claim 1, further comprising an image output unit that outputs a fisheye image in the cutout region.   3. The peripheral blur amount calculation unit calculates a blur amount of a peripheral portion of the fisheye image based on a blur amount detected at a central portion of the fisheye image. The fish-eye image processing apparatus described in 1. The peripheral blur amount calculating unit
A first conversion unit that generates a perspective projection image from the fisheye image based on projective conversion information;
A second conversion unit that reversely converts the perspective projection image into a fisheye image based on the projective conversion information;
With
The blur analysis unit calculates a blur amount of the entire fisheye image based on the perspective projection image generated by the first conversion unit,
The second conversion unit performs back projection conversion on the total amount of blur of the fisheye image calculated by the blur analysis unit based on the projective transformation information, and calculates the amount of blur at the periphery of the fisheye image. The fish-eye image processing apparatus according to claim 1 or 2. The fisheye optical system has a zoom function,
The fisheye according to any one of claims 1 to 4, wherein the cutout region determination unit changes the cutout region nonlinearly with respect to a change in a focal length of the fisheye optical system. Image processing device.   The fisheye image processing apparatus according to any one of claims 1 to 5, wherein the fisheye image includes a moving image. An imaging unit that captures a fisheye image using a fisheye optical system;
A fisheye image processing apparatus according to any one of claims 1 to 6,
A fish-eye image capturing apparatus comprising: A fisheye image processing method for correcting distortion of a fisheye image captured using a fisheye optical system,
A detection step for detecting a blur amount of the fisheye image;
A calculation step of calculating a blur amount of a peripheral portion of the fish-eye image based on the blur amount detected in the detection step;
A determination step of determining a cut-out region of the fish-eye image based on the calculation result in the calculation step;
A fish-eye image processing method comprising:

Images