JP2020123857A - Imaging apparatus, image correction method, and program - Google Patents

Abstract

To provide an imaging apparatus that can prevent a reduction in image quality even when a direction of reading out the output of pixels from an image pick-up device is changed.SOLUTION: An imaging apparatus has an image pick-up device 130 from which the output of pixels is read out from top to bottom or bottom to top, and comprises: the image pick-up device that has a first light-shielding area 11 having pixels the output of which is not read out according to whether the output of pixels is read out from top to bottom or read out from bottom to top, a second light-shielding area 12 having pixels the output of which is read out when the output of pixels is read out from top to bottom or read out from bottom to top, and an effective area 13 that outputs image data for incident light; storage means for defective pixel information in which the coordinates of a defective pixel are registered; and image correction means that excludes the defective pixel registered in the defective pixel information from the second light-shielding area, determines a black level from the output of pixels from the second light-shielding area, and corrects an image in the effective area based on the black level.SELECTED DRAWING: Figure 10

Description

The present invention relates to an imaging device, an image correction method, and a program.

In order to reduce the influence on the image of the change in black level due to dark current in the image pickup apparatus, there is a method in which a light-shielding area (optical black) is provided and used as a reference for the black level.

In addition, the image sensor may have a defective pixel as the resolution of the image sensor increases, and conventionally, the coordinates of the defective pixel are registered before shipment, and are registered when the user performs an image capturing operation. A method of correcting the output of the pixel at the given coordinates is adopted. Since defective pixels also occur in the light-shielded area, the output of the defective pixel registered in advance is not used for determining the black level when determining the black level from the output of the light-shielded area.

By the way, an image pickup apparatus capable of picking up a wide angle of view may be equipped with a plurality of image pickup elements. However, in order to ensure the quality of joints of image data picked up by the plurality of image pickup elements, the image pickup elements output pixels. There has been devised a technique for changing the reading direction of (see Patent Document 1, for example). Patent Document 1 discloses an imaging device that corrects the exposure timing deviation of the imaging element in order to ensure the quality of the joint.

However, when the pixel output read direction changes depending on the image sensor, the coordinates of the defective pixel registered in advance and the coordinates of the defective pixel read from the light-shielding area may not match. There is a problem that the number of pixels that can be used for black level correction is reduced. If the number of pixels that can be used to correct the black level decreases, the accuracy of the black level may decrease and the image quality may deteriorate.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an imaging device capable of suppressing deterioration of image quality even when the output reading direction of pixels of an imaging element is changed.

In view of the above problems, the present invention is an imaging device having an imaging device in which pixel outputs are read from top to bottom or from bottom to top, and pixel outputs are read from top to bottom, or from bottom. A first light-shielding region having a pixel whose output is not read, depending on whether to read upward, a second light-shielding region having a pixel whose output is read even if the pixel output is read from top to bottom or from bottom to top An image sensor having a light-shielding area and an effective area for outputting image data for incident light, a storage unit for storing defective pixel information in which the coordinates of the defective pixel are registered, and the defective pixel registered in the defective pixel information. Is excluded from the second light-shielding region, the black level is determined from the output of the pixel of the second light-shielding region, and the image correction unit that corrects the image of the effective region based on the black level is included. Characterize.

It is possible to provide an image pickup apparatus capable of suppressing deterioration in image quality even if the reading direction of the pixel output of the image pickup element changes.

It is an example of the figure which expands and shows an image sensor and the 1st shade area provided in an image sensor. It is an example of a sectional view of an imaging device. It is an example of the hardware block diagram of an imaging device. FIG. 3 is an example of a functional block diagram showing, in block form, the functions of the imaging device. It is an example of a flow chart showing the imaging procedure performed by the imaging device. It is an example of the figure explaining the projection relation in an imaging device. It is an example of the figure explaining the data structure of the image data of a spherical image. It is an example of a diagram illustrating a conversion parameter used by the imaging device. It is an example of the figure which shows the omnidirectional image which an imaging device produces|generates from two partial images. It is a figure which shows the example of arrangement|positioning of the light-shielding area and effective area|region provided in the image sensor. It is a figure which shows an example of the coordinate of the defective pixel of the 2nd shade area in non-inversion reading and inversion reading. FIG. 6 is an example of a flowchart showing a procedure in which the image correction unit uses the output of the light-shielded region to suppress the image quality deterioration of the effective region due to dark current. FIG. 6 is an example of a diagram illustrating reduction of the influence of dark current correction due to saturation of the effective area. FIG. 6 is an example of a flowchart showing a procedure in which the image correction unit uses the output of the light-shielded region to suppress the deterioration of the image quality of the effective region due to the dark current even when the saturated region occurs.

Hereinafter, as an example of a mode for carrying out the present invention, an image pickup apparatus and an image correction method performed by the image pickup apparatus will be described with reference to the drawings.

<Supplementary information on conventional technology>
First, the conventional technique will be supplemented with reference to FIG. FIG. 1 is an enlarged view showing the image sensor 130 and the first light-shielding region 11 provided in the image sensor 130.

Since pixel defects tend to occur frequently during long-time imaging, which leads to deterioration in image quality, it is common to register the coordinates of defective pixels in the adjustment process before shipping. At the time of image capturing, the output of the pixel (defective pixel) at the registered coordinates is corrected by the output of the surrounding pixels. In addition, since the defective pixel also exists in the first light-shielding region 11 and causes an error in the output (detection) value of the first light-shielding region 11, the coordinates of the defective pixel in the first light-shielding region 11 are also registered in the adjustment process. ing. When determining the black level at the time of image capturing, the output of the pixel (defective pixel) at the registered coordinates is not used for determining the black level (hereinafter, simply referred to as “removal”).

By the way, in an image pickup apparatus having a plurality of image pickup elements 130 as shown in FIG. 1, it is necessary to join images picked up by each of the plurality of image pickup elements 130 to generate one image. It is said that it is desirable to match the exposure timing of the joint portion by making the reading directions different among the plurality of image pickup devices 130. By doing so, it is possible to reduce the deviation of the imaging time of the joint portion of different image pickup devices 130 and improve the image quality of the joint portion.

For example, as a method of aligning the reading directions, it is conceivable to prepare a plurality of physically different image pickup device substrates (substrates on which the image pickup device 130 is mounted), but in this case, different types of image pickup device substrates are required and the cost is increased. Will increase. Therefore, in order to reduce the cost, one type of image pickup device substrate is used, and the read direction from the image pickup device 130 (whether the pixels in line units are read from above or below) is operated in the opposite direction to change the read direction. A method of adjusting may be adopted.

The image sensor 130 of FIG. 1 is read from the top to the bottom (non-reverse reading direction) or from the bottom to the top (reverse reading direction). The reading direction is determined and the coordinate of the defective pixel is registered. The order was not always fixed. That is, the coordinates of the defective pixel are not necessarily registered in advance in the state where the reading direction is fixed. Therefore, the following inconvenience occurs.

FIG. 1A shows the physical arrangement of the first light shielding area 11 and the difference between the reading areas depending on the reading direction. The area surrounded by the solid line is the non-inverted read area 35 read in the non-inverted read direction, and the area surrounded by the dotted line is the inverted read area 34 read in the inverted read direction. As shown, the non-inverted read region 35 and the inverted read region 34 do not completely match. Therefore, the upper band 33 of the non-inverted read region 35 is not read in the reverse read direction, and the lower band 32 of the reverse read region 34 is not read in the non-inverted read direction.

This is because it is not necessary to read all the pixels of the first light-shielding area 11, the height of the first light-shielding area 11 is not always constant, or the image data is not viewed by the user, and thus the image is inverted. The reason is that it is only necessary to read a part. As will be described later, when the effective area 13 (non-light-shielded area) on which the image of the subject is projected is read out in reverse, the image data is turned upside down, so a process called upside down is performed.

The overlapping area of the non-inverted read area 35 and the inverted read area 34 is referred to as a match area 31 because there are defective pixels that match the coordinates of the defective pixels registered in both the non-inverted read and the inverted read.

FIG. 1B shows an inverted read area 34 that has been inverted read and defective pixels 41a to 41d, 42a, and 42b. Since reverse reading is performed, the lower band portion 32 of FIG. 1A is on the upper side and the matching region 31 is on the lower side. Defective pixels exist in the lower band portion 32 and the matching area 31, respectively.

FIG. 1C shows the non-inverted read area 35 and the defective pixel which are non-inverted read. Since non-inverted reading is performed, the upper band portion 33 of FIG. 1A is on the upper side and the matching region 31 is on the lower side. Defective pixels exist in the upper band portion 33 and the matching area 31, respectively.

Comparing FIG. 1B and FIG. 1C, the defective pixels 41a to 41d existing in the matching area 31 are present in line symmetrical positions above and below the matching area 31 even if the reading direction is changed. When the region 31 is folded horizontally, there is a defective pixel in the matching region 31 in another reading direction). That is, if the coordinates of the defective pixels 41a to 41d in the matching area 31 are registered in advance, the coordinates are registered regardless of whether the read direction determined for maintaining the image quality of the joint is the non-reversed read direction or the reversed read direction. Therefore, it can be excluded from the determination of the black level at the time of imaging.

However, the defective pixel of the lower band portion 32 cannot be used for image quality correction at the time of imaging when the read direction determined for maintaining the image quality at the joint is the non-reverse read direction. The defective pixel of the upper band 33 cannot be used for image quality correction at the time of imaging when the read direction determined to maintain the image quality of the joint is the reverse read direction.

As described above, when the reading direction in the process of registering the coordinates of the defective pixel does not match the reading direction in the state shipped as the imaging device, the image data read from the first light-shielding area 11 has the coordinates of the defective pixel registered. Since the unshielded region may be included, as a result, the region that can be used as the first light shielding region 11 is only the coincidence region 31, and the accuracy of the black level is deteriorated due to the decrease in the number of samples in the light shielding region. there were.

<About terms>
The reading direction of the pixel output refers to whether the output for one line is sequentially read from the top to the bottom of the image sensor or from the bottom to the top.

The effective area for outputting image data with respect to the incident light refers to an area of pixels that is not shielded.

Image correction means changing to a value close to true, changing to an appropriate pixel value, and improving image quality.

<Structure example>
Hereinafter, the overall configuration of the imaging device 100 according to the present embodiment will be described with reference to FIGS. 2 and 3. FIG. 2 is a sectional view of the image pickup apparatus 100 according to the present embodiment. The image pickup apparatus 100 shown in FIG. 2 includes an image pickup body 19, a casing 17 that holds the image pickup body 19 and other components such as a controller board and a battery, and an image pickup button 18 provided on the casing 17.

The image pickup body 19 shown in FIG. 2 has two image forming optical systems 20A and 20B and an image pickup element 130A and an image pickup element 130B (any image pickup element of the image pickup elements 130A and 130B is set as the image pickup element 130). .. The image pickup devices 130A and 130B are CCD (Charge Coupled Device) sensors, CMOS (Complementary Metal Oxide Semiconductor) sensors, and the like. The image forming optical systems 20A and 20B are configured, for example, as fisheye lenses of 7 elements in 6 groups. In the embodiment shown in FIG. 2, the fish-eye lens has a total angle of view larger than 180 degrees (=360 degrees/n; the number of optical systems n=2), and preferably has a field angle of 190 degrees or more. .. A combination of such wide-angle imaging optical systems 20A and 20B and one image pickup device 130 is called a wide-angle image pickup optical system.

The positions of the optical elements (lens, prism, filter, and aperture stop) of the two imaging optical systems 20A and 20B are determined with respect to the image pickup elements 130A and 130B. The optical axes of the optical elements of the imaging optical systems 20A and 20B are positioned orthogonal to the center of the light receiving areas of the corresponding image pickup elements 130A and 130B, and the light receiving areas are the image forming planes of the corresponding fisheye lenses. Positioning is performed so that

In the embodiment shown in FIG. 2, the image forming optical systems 20A and 20B have the same specifications and are combined in opposite directions so that their optical axes coincide with each other. The image pickup devices 130A and 130B convert the received light distribution into image signals and sequentially output image frames to the image processing blocks on the controller board. Although details will be described later, the images captured by the image pickup devices 130A and 130B are combined to generate an image with a solid angle of 4π stearadians (hereinafter referred to as a “spherical image”). It The omnidirectional image is an image of all directions that can be overlooked from the imaging point. In the embodiment to be described, it is assumed that a spherical image is generated. However, a so-called panoramic image obtained by capturing a wide-angle image (for example, 90 degrees to 360 degrees) only on a horizontal plane may be used, and a panoramic view of the spherical surface. It may be an image obtained by capturing a part of the above. Further, the spherical image can be saved as a still image or a moving image.

FIG. 3 shows a hardware configuration of the image pickup apparatus 100 according to the present embodiment. The image pickup apparatus 100 is connected via a CPU (Central Processing Unit) 112, a ROM (Read Only Memory) 114, an image processing block 116, a moving image compression block 118, and a DRAM (Dynamic Random Access Memory) interface 120. The DRAM 132, the external storage 134 connected via the external storage interface 122, the attitude sensor 136 connected via the external sensor interface 124, the USB connector 138 connected via the USB interface 126, and the serial block 128. A wireless NIC (Network Interface Card) 140 and a video output interface 129 which are connected to each other via the.

The CPU 112 controls the operation and overall operation of each unit of the imaging device 100. The ROM 114 stores a control program (an example of a program in the claims) written in a code readable by the CPU 112 and various parameters. The image processing block 116 is connected to the image pickup device 130A and the image pickup device 130B, and the image signal of the image picked up by each of them is input. The image sensor 130A has a first temperature sensor 150A that detects the temperature of the image sensor 130A, and the image sensor 130B has a second temperature sensor 150B that detects the temperature of the image sensor 130B.

The image processing block 116 is configured to include an ISP (Image Signal Processor) and the like, and performs shading correction, Bayer interpolation, white balance correction, gamma correction, and the like on the image signal input from the image sensor 130. The image processing block 116 further performs a process of synthesizing a plurality of images acquired from the image pickup device 130A and the image pickup device 130B through the above process, thereby performing the process of generating the above-described spherical image.

The moving picture compression block 118 is an MPEG-4 AVC/H. It is a codec block that performs video compression and decompression such as H.264. The moving image compression block 118 is used to store the generated moving image data of the spherical image, or to reproduce and output the stored moving image data. The DRAM 132 provides a storage area for temporarily storing data when performing various kinds of signal processing and image processing.

The attitude sensor 136 includes an acceleration sensor, a gyro sensor, a geomagnetic sensor, or a combination thereof, and is used to specify the attitude of the imaging device 100. For example, a triaxial acceleration sensor can detect a triaxial acceleration component. The triaxial gyro sensor can detect triaxial angular velocities. The geomagnetic sensor can measure the direction of the magnetic field. The outputs of these sensors are used alone or in combination to provide three attitude angles of the image pickup apparatus 100. The information obtained from the attitude sensor 136 is used to perform the zenith correction of the omnidirectional image, and can also be used to perform the image rotation processing according to the attention point described later.

An external storage 134 such as a memory card inserted in a memory card slot is connected to the external storage interface 122. The external storage interface 122 controls reading and writing with respect to the external storage 134.

A USB connector 138 is connected to the USB interface 126. The USB interface 126 controls USB communication with an external device such as a personal computer connected via the USB connector 138. The serial block 128 controls serial communication with an external device such as a personal computer, and is connected to the wireless NIC 140. The video output interface 129 is an interface for connecting to an external display such as HDMI (High-Definition Multimedia Interface, HDMI is a registered trademark), and an image before or during recording or a recorded image is displayed on the external display. You can output the video to.

Although the USB connector 138, the wireless NIC 140, and the video output interface 129 based on HDMI (registered trademark) are shown as an example, the invention is not limited to a specific standard. In other embodiments, wired communication such as wired LAN (Local Area Network), other wireless communication such as Bluetooth (registered trademark) and wireless USB, other video such as DisplayPort (registered trademark) and VGA (Video Graphics Array). It may be connected to an external device via the output interface.

When the power is turned on by operating the power switch, the control program is loaded into the main memory. The CPU 112 controls the operation of each part of the device according to the control program read into the main memory, and temporarily stores the data necessary for the control in the memory. Thereby, each functional unit and processing of the image capturing apparatus 100, which will be described later, are realized.

<About functions>
FIG. 4 is an example of a functional block diagram showing the functions of the image capturing apparatus 100 in block form. The imaging apparatus 100 includes a captured image acquisition unit 202, an image correction unit 203, a stitching processing unit 204, a zenith correction unit 206, a celestial sphere image generation unit 208, and an image compression unit 210.

The image pickup apparatus 100 also has a defective pixel DB 305 built in the ROM 114 or the like. Table 1 shows the coordinates of defective pixels registered in the defective pixel DB 305.

欠陥画素ＤＢ３０５には例えば座標順に欠陥画素の座標が登録されている。欠陥画素ＤＢ３０５は欠陥画素の座標が登録された欠陥画素情報の記憶手段である。欠陥画素とは正常画素と特性が異なる画素（典型的な例として白傷、黒傷がある）をいう。欠陥画素は撮像された画像の劣化をもたらすため、撮像装置は撮像された有効領域の画像を補正することが一般的である。なお、欠陥画素ＤＢ３０５には有効領域の欠陥画素の座標と、第一遮光領域の欠陥画素の座標と、第二遮光領域の欠陥画素の座標が登録されている。それぞれが区別して登録されていてもよい。区別されていないとしても有効領域、第一遮光領域及び第二遮光領域の座標が既知なので、欠陥画素ＤＢ３０５からそれぞれの欠陥画素を特定できる。

In the defective pixel DB 305, the coordinates of defective pixels are registered in the order of coordinates, for example. The defective pixel DB 305 is a storage unit of defective pixel information in which the coordinates of defective pixels are registered. A defective pixel is a pixel having characteristics different from those of a normal pixel (typical examples include white scratches and black scratches). Since the defective pixel causes deterioration of the captured image, the image capturing device generally corrects the captured image of the effective region. In the defective pixel DB 305, the coordinates of the defective pixels in the effective area, the coordinates of the defective pixels in the first light shielding area, and the coordinates of the defective pixels in the second light shielding area are registered. Each may be registered separately. Even if they are not distinguished, the coordinates of the effective area, the first light shielding area, and the second light shielding area are known, so that each defective pixel can be specified from the defective pixel DB 305.

(Explanation of each function)
The captured image acquisition unit 202 controls the above-described image sensor 130A and image sensor 130B and acquires captured images from each of them. In the case of a still image, two captured images for one frame acquired at the timing when the shutter is pressed are acquired. In the case of a moving image, consecutive frames are sequentially captured, and two captured images are acquired for each frame. The image picked up by each of the image pickup devices 130 is a fish-eye image in which a hemisphere of the whole celestial sphere is included in the field of view, and constitutes a partial image of the whole celestial sphere image. Hereinafter, the image captured by each of the image sensors 130 may be referred to as a partial image.

The image correction unit 203 (an example of an image correction unit) acquires the coordinates of the defective pixel from the defective pixel DB 305 and corrects or excludes the pixel value of the defective pixel. In the effective area 13, the pixel value of the defective pixel is mainly corrected by using the pixel values of the pixels around the defective pixel. In the first light shielding area 11 and the second light shielding area 12, defective pixels are excluded from the pixels for determining the black level. In order to eliminate the inconvenience described with reference to FIG. 1, the image correction unit 203 of this embodiment determines the black level by using the output of the pixel of the second light shielding area 12.

The stitching processing unit 204 (an example of stitching means) detects the stitching position between the two acquired partial images, and executes processing for stitching the two partial images. In the joint position detection process, a process of detecting the position shift amount of each of the plurality of corresponding points in the overlapping region existing between the plurality of partial images is performed for each frame.

The zenith correcting unit 206 controls the posture sensor 136 shown in FIG. 3 to detect the posture angle of the imaging device 100, and to make the zenith direction of the generated omnidirectional image coincide with the predetermined reference direction. The correction process of is executed. Here, the predetermined reference direction is typically a vertical direction, and is a direction in which gravitational acceleration acts. By correcting the zenith direction of the omnidirectional image so as to match the vertical direction (the zenith direction), especially in a moving image, even if the field of view is changed at the time of browsing, the user does not feel uncomfortable such as three-dimensional sickness It becomes possible to do.

The omnidirectional image generation unit 208 executes a process of generating an omnidirectional image from the two captured partial images in a state in which the processing results of the joining processing unit 204 and the zenith correction unit 206 are reflected. In the embodiment to be described, there is a conversion parameter for generating the omnidirectional image from the two partial images, and the joint processing unit 204 reflects the result of joint position detection on this conversion parameter. .. The zenith correcting unit 206 also reflects the result of the zenith correction on the conversion parameter. Then, the celestial sphere image generation unit 208 generates a celestial sphere image from the two partial images by using the conversion parameters in which these processing results are reflected. By processing in this way, the processing load for obtaining the final spherical image can be reduced.

However, the zenith correction is not limited to the above-described embodiment, and the two partial images are joined to generate a omnidirectional image, and the generated omnidirectional image is subjected to the zenith correction process. It can also be configured to generate a rendered spherical image. The conversion parameters will be described later.

The image compression unit 210 is configured to include a still image compression block, and when capturing a still image, compresses the captured image into image data in a predetermined still image format such as JPEG (Joint Photographic Experts Group). The image compression unit 210 also controls the moving image compression block 118 shown in FIG. 3, and when capturing a moving image, it compresses consecutive captured image frames into image data in a predetermined moving image format. The moving image compression format is not particularly limited, but H.264. H.264/MPEG-4 AVC (Advanced Video Coding), H.264. Various moving image compression formats such as H.265/HEVC (High Efficiency Video Coding), Motion JPEG, and Motion JPEG2000 can be cited. The generated image data is stored in the external storage 134 attached to the image capturing apparatus 100 or other storage such as a flash memory included in the image capturing apparatus 100.

<Overall procedure>
FIG. 5 is an example of a flowchart showing an image capturing procedure executed by the image capturing apparatus 100. Hereinafter, the processing executed by the imaging device 100 will be described with reference to FIG. Note that the processing shown in FIG. 5 is started in response to a recording instruction such as pressing of the imaging button by the user. The process shown in FIG. 5 describes the case of capturing a still image.

In step S101, the captured image acquisition unit 202 acquires captured images from the image sensors 130A and 130B. The captured image will be described with reference to FIGS. 6 to 8.

FIG. 6 is a diagram illustrating a projection relationship in the image pickup apparatus 100 according to the present embodiment. In the present embodiment, an image picked up by one fisheye lens is an image picked up in a hemispherical direction from an image pickup point. Further, as shown in FIG. 6A, the fisheye lens generates an image with an image height h corresponding to the incident angle φ with respect to the optical axis. The relationship between the image height h and the incident angle φ is determined by a projection function according to a predetermined projection model.

The projection function depends on the nature of the fisheye lens. The projection model includes an equidistant projection method (h=f·φ), a central projection method (h=f·tan φ), a stereoscopic projection method (h=2f·tan(φ/2)), an equisolid angle projection method. (H=2f·sin(φ/2)) and the orthogonal projection method (h=f·sinφ). In either method, the image height h of the formed image is determined according to the incident angle φ from the optical axis and the focal length f. Further, in the present embodiment, it is assumed that a so-called circumferential fisheye lens having a smaller image circle diameter than the diagonal line of the image is adopted, and the partial image obtained is approximately a hemisphere of the imaging range as shown in FIG. 6B. It becomes a planar image including the entire image circle onto which the minutes are projected.

FIG. 7 is a diagram for explaining the data structure of the image data of the spherical image used in this embodiment. Note that FIG. 7 shows a state in which the images of two hemispheres have already been joined. As shown in FIG. 7B, the spherical image format has a pixel value in which an angle φ corresponding to an angle formed with respect to a predetermined axis and a horizontal angle θ corresponding to a rotation angle around the axis are coordinates. Expressed as an array of. As shown in FIG. 7A, each coordinate value (θ, φ) is associated with each point on the spherical surface representing the omnidirectional direction centered on the imaging point, and the omnidirectional direction is on the omnidirectional image. Is mapped to.

FIG. 8 is a diagram for explaining conversion parameters used by the image pickup apparatus according to the present embodiment. The conversion parameter defines the projection from the partial image represented by the plane coordinate system to the image represented by the spherical coordinate system. As shown in FIGS. 8A and 8B, the conversion parameters are, for each fish-eye lens, the coordinate values (θ, φ) of the corrected image and the pre-correction mapped to the coordinate values (θ, φ). The information for associating with the coordinate values (x, y) of the partial image is stored for all coordinate values (θ, φ). In the example of FIG. 8, the angle that one pixel is in charge of is 1/10 degrees in both the φ direction and the θ direction, and the conversion parameter has information indicating the correspondence of 3600×1800 for each fisheye lens. .. The original conversion parameters may be calculated in advance after correcting the distortion from the ideal lens model by the manufacturer or the like and tabulated.

It returns to FIG. 5 and demonstrates. In step S102, the image correction unit 203 corrects the images captured by the image sensor 130A and the image sensor 130B, respectively. Details of this processing will be described later.

Next, in step S103, the joining processing unit 204 detects the joining position in the acquired overlapping area between the two partial images, and reflects the result of the joining position detection on the conversion parameter. By reflecting the result of the joint position detection, the conversion parameter shown in FIG. 8A shows that the coordinate values (x, y) of the partial image reflecting the joint position correction with respect to the coordinate value (θ, φ) of the corrected image. Are modified so that

In step S104, the zenith correction|amendment part 206 detects the attitude|position angle with respect to the gravity direction of the imaging device 100, and correct|amends a conversion parameter so that the zenith direction of the generated omnidirectional image may correspond to a vertical direction. The zenith correction can be performed by the same processing as the three-dimensional rotation conversion described in detail later, but the details will not be described here.

In step S105, the omnidirectional image generation unit 208 uses the conversion parameter to execute a process of generating an omnidirectional image from the two captured partial images. In step S105, the partial image is first converted from the plane coordinate system to the spherical coordinate system using the conversion parameter. Then, the partial images of the two spherical coordinate systems are image-synthesized to generate a spherical image.

FIG. 9 shows a celestial sphere image generated from the two partial images by the image pickup apparatus 100 according to the present embodiment. 9A is a hemispherical image captured by the image capturing apparatus 100 (front side), FIG. 9B is a hemispherical image captured by the image capturing apparatus 100 (rear side), and FIG. 9C is an equirectangular projection method. It is the figure which showed the represented image (henceforth "equal cylinder projection image").

As shown in FIG. 9A, the image obtained by the image sensor 130A becomes a hemispherical image (front side) curved by the fisheye lens 102a described later. Further, as shown in FIG. 9B, the image obtained by the image sensor 130B becomes a hemispherical image (rear side) curved by the fisheye lens 102b described later. Then, the hemisphere image (front side) and the hemisphere image inverted by 180 degrees (rear side) are combined to create an equirectangular cylindrical projection image EC as shown in FIG. 9C.

By using OpenGL ES (Open Graphics Library for Embedded Systems) etc., the equirectangular projection image is pasted so as to cover the spherical surface, and a spherical image is created. In this way, the omnidirectional image is represented as an image in which the equirectangular cylindrical projection image EC faces the center of the sphere. OpenGL ES is a graphics library used for visualizing 2D (2-Dimensions) and 3D (3-Dimensions) data. The spherical image may be a still image or a moving image.

It returns to FIG. 5 and demonstrates. In step S106, the image compression unit 210 performs image compression on the generated image data of the spherical image, and in step S107, the generated image data is stored in the storage.

<Regarding the light shielding area in the image sensor>
Next, the light blocking area of the image sensor 130 will be described with reference to FIG. FIG. 10 is a diagram showing an arrangement example of the light shielding areas and the effective area 13 provided in the image pickup devices 130A and 130B. Since the two image sensors 130A and 130B may have the same structure, only one image sensor 130A or 130B is shown in the figure.

In an image sensor such as a CMOS or a CCD, an output due to a dark current may be generated even when light is not incident on a pixel at the time of high gain, long-time imaging, or high temperature. The image may appear as if it were dark (a part that should be dark appears bright). In order to suppress image quality deterioration due to dark current, the image pickup devices 130A and 130B have one or more light shielding regions. In FIG. 10, a first light blocking area 11 and a second light blocking area 12 are provided. The first light-shielding region 11 at the upper end may be referred to as a top light-shielding region, and the second light-shielding region 12 at the lateral (left) end may be referred to as a side light-shielding region. The effective area 13 outputs the image data for the incident light (outputs the image data seen by the user).

Since the pixels in the light-shielded area also output a dark current that is close to (comparable to) the pixels in the effective area 13, the image correction unit 203 detects (detects) the output of the pixel in the light-shielded area and shifts from the effective area 13 to the light-shielded area. By subtracting the output, only the output of the pixel in the effective area 13 with respect to the incident light is extracted. More specifically, the dark current correction is performed by obtaining the average value of the output of the light-shielded area at the time of image capturing to obtain the black level, and subtracting this average value (black level) from the output of each pixel of the effective area 13.

Since noise is also included in the output of the pixels in the light-shielded area, a stable black level can be calculated as the number of samples (the number of pixels in the light-shielded area) increases. As described with reference to FIG. 1, the number of samples (the number of pixels) in the first light-shielding region 11 may become insufficient, so the second light-shielding region 12 is used for determining the black level in this embodiment. It is not always necessary to use all of the second light-shielding region 12, and for example, only the output of some pixels corresponding to the size of the first light-shielding region 11 may be used. The second light shielding area 12 may be on the right side of the image pickup devices 130A and 130B.

Further, the image sensor 130 has column parallel ADCs 15 on the top and bottom (or one side). The column parallel ADC 15 is an A/D converter that reads out the output of each column in parallel for each horizontal line. In the present embodiment, the output of each pixel is read for each line in the horizontal direction either from top to bottom (non-reverse reading) or from bottom to top (reverse reading).

<Regarding the coordinates of the defective pixel in the second light shielding area>
Although it has been described that the number of registered defective pixels (matching region 31) that can be used during imaging is small in the first light-shielding region 11, the second light-shielding region 12 does not have such an inconvenience.

FIG. 11 shows an example of the coordinates of the defective pixel in the second light-shielding region 12 in the non-inverted reading and the inverted reading. FIG. 11A shows an output image by non-reverse reading, and FIG. 11B shows an output image by reverse reading. FIG. 11 schematically shows a case where the character “F” is picked up. In FIG. 11B, it can be seen that the “F” is inverted because it is read from bottom to top.

The defective pixels 51a to 51d in the second light shielding area 12 are also inverted vertically, but the defective pixels 51a to 51d in the second light shielding area 12 are on the same horizontal line as the effective area 13. Therefore, the output of the second light-shielding region 12 is also read together with the effective region 13, and there are no or very few pixels (pixels whose coordinates do not match between non-inverted read and inverted read) depending on the read direction. Even if there are some ring pixels (reasonable and minimum number of pixels around the effective area 13 required for filtering) and Bayer array area mismatch (pixel mismatch due to Bayer filter and pixel shift), the effective area is small. It is negligible compared to the 13 pixels. Therefore, if the image correction relating to the dark current is performed using the second light shielding region 12, the inconvenience described in FIG. 1 can be suppressed.

<Procedure for image correction>
FIG. 12 is an example of a flowchart showing a procedure in which the image correction unit 203 suppresses the image quality deterioration of the effective region 13 due to the dark current by using the output of the light shielding region.

The image correction unit 203 extracts the output of the first light shielding area 11 (S11). In the case of non-inverted read, the pixels in the non-inverted read area 35 are taken out, and in the case of inverted read, the pixels in the inverted read area 34 are taken out.

The image correction unit 203 extracts the output of the matching area 31 (S12). This is because only the coordinates of the defective pixels in the matching area 31 may be registered in the defective pixel DB 305. It is assumed that the coordinates of the matching area 31 are registered in the image correction unit 203 in advance. If not registered, about 2/3 from the bottom of the pixels of the first light shielding area 11 read in step S11 may be regarded as the matching area 31. This is because, as shown in FIG. 1B or FIG. 1C, the coincidence region 31 exists below the first light-shielding region 11 in both non-inversion reading and inversion reading.

Next, the image correction unit 203 reads the coordinates of the defective pixel in the matching area 31 from the defective pixel DB 305 (S13). A defective pixel having coordinates in the pixel range of the matching area 31 may be identified from the defective pixel DB 305. Then, the defective pixel is excluded from the pixels in the matching area 31 (S14). Exclusion means not being used to calculate the average output from the pixels.

Next, the image correction unit 203 extracts the output of the second light shielding area 12 (S15). The second light-shielding region 12 does not change even if the reading direction changes.

Then, the image correction unit 203 reads the coordinates of the defective pixel in the second light-shielding area 12 from the defective pixel DB 305 (S16). The defective pixel having the coordinates of the pixel range of the second light shielding area 12 may be specified from the defective pixel DB 305.

Next, the image correction unit 203 excludes the defective pixel from the pixels of the second light shielding area 12 (S17). Exclusion means not being used to calculate the average output from the pixels.

The image correction unit 203 calculates the average (black level) of the outputs of the pixels of the matching area 31 excluding the defective pixels and the pixels of the second light shielding area 12 (S18). Thereby, the black level can be determined excluding the output of the defective pixel. Further, the influence of the dark current on the output of the effective area 13 can be estimated.

Next, the image correction unit 203 subtracts the average calculated in step S18 from the output of the pixels in the effective area 13 (S19). As a result, it is possible to suppress deterioration of image quality in which the originally dark area of the image data of the effective area 13 looks bright due to the dark current.

In the process of FIG. 12, the first light-shielding region 11 is used for dark current correction, but when the second light-shielding region 12 is used, the number of samples is sufficient, so the first light-shielding region 11 is used for dark current correction. You don't have to.

<Reduction of influence of dark current correction due to saturation of effective area>
FIG. 13 is a diagram for explaining reduction of the influence of dark current correction due to saturation of the effective area 13. It is known that when intense light is incident on an arbitrary pixel in the effective area 13, this pixel is saturated. Saturation refers to charging up to the upper limit of the capacity of the capacitor that stores the charge generated in the pixel. The saturated pixel area is referred to as a saturated area 53.

Since the second light-shielding region 12 exists on the same horizontal line as the effective region 13, when the saturated region 53 occurs in the effective region 13, the pixel 54 of the second light-shielding region 12 existing on the same horizontal line. Output may also be affected. The pixels in the second light-shielding region 12 may accumulate the charges flowing out of the saturation region 53, in addition to the dark current, even though they are shielded from light.

In the wide-angle imaging device capable of imaging the celestial sphere as in the present embodiment, although the imaging region is wide, the area of the saturation region 53 is small, and the influence is considered to be slight, but the influence can be eliminated as much as possible. preferable.

Therefore, the image correction unit 203 excludes the pixel 54 of the second light-shielding region 12 existing on the same horizontal line as the saturation region 53 from the target of black level determination as the detection exclusion region. By doing so, it is possible to suppress deterioration of the dark current correction effect even if the saturated region 53 occurs in the effective region 13. It should be noted that the pixel value of the actual image (for example, the pixel showing the pixel value of 255 when the pixel value of 0 to 255 is taken) may be used for the detection of the saturated region 53, or the evaluation value by AE (Automatic Exposure). The data of the pixel block used for the calculation may be used. The AE is a function of calculating an evaluation value of the brightness value from the output of the image sensor 130 and controlling the gain, aperture, and exposure time (shutter speed) so that an appropriate exposure amount can be obtained based on the evaluation value.

FIG. 14 is an example of a flowchart showing a procedure in which the image correction unit 203 suppresses the image quality deterioration of the effective area 13 due to the dark current by using the output of the light shielding area even when the saturated area 53 occurs. In the description of FIG. 14, differences from FIG. 12 will be mainly described. First, the processes of steps S11 to S17 may be the same as in FIG.

In step S17-2, the image correction unit 203 determines whether or not there is a saturated region (S17-2). Then, if there is a saturated region, pixels of the same horizontal line as the saturated region are excluded from the pixels of the second light shielding region 12 (S17-3).

By so doing, it is possible to prevent the effect of dark current correction from decreasing even if a saturated region occurs in the effective region 13.

<Summary>
As described above, the image pickup apparatus 100 according to the present embodiment performs the dark current correction using the second light-shielding region 12 (side light-shielding region), and thus the image pickup elements having different read directions for connecting a plurality of image data. Therefore, even if the number of samples in the first light-shielding region 11 is small, the black level can be accurately determined, and the deterioration of image quality can be suppressed.

<Other application examples>
The best mode for carrying out the present invention has been described above with reference to the embodiments, but the present invention is not limited to these embodiments, and various modifications can be made without departing from the gist of the present invention. And substitutions can be added.

For example, in the present embodiment, the image is corrected before joining the two images, but the image may be corrected after joining.

Further, in the present embodiment, the correction is performed by the image capturing apparatus 100, but the system from image capturing to image correction may be performed. For example, the imaging device 100 transmits two images and imaging conditions to the smartphone or the server, and the smartphone or server returns the corrected two image data to the imaging device 100.

Further, the configuration example of FIG. 4 shown in the above embodiment is divided according to the main function in order to facilitate understanding of the processing of the image pickup apparatus 100. However, the present invention is not limited by the division method or name of each processing unit. The image pickup apparatus 100 can be divided into more processing units according to the processing content. Also, one processing unit can be divided so as to include more processing.

11 First Light-Shielding Area 12 Second Light-Shielding Area 13 Effective Area 31 Matching Area 34 Inversion Reading Area 35 Non-Inversion Reading Area 100 Imaging Device

JP, 2008-137592, A

Claims (9)

An image pickup device having an image pickup device in which the output of a pixel is read from top to bottom or from bottom to top,
A first light-shielding region having a pixel whose output is not read by reading the output of the pixel from top to bottom or from bottom to top;
Whether the pixel output is read from top to bottom or from bottom to top, a second light-shielding region having a pixel whose output is read, and
An image pickup device having an effective area for outputting image data for incident light;
Storage means for defective pixel information in which coordinates of defective pixels are registered;
The defective pixel registered in the defective pixel information is excluded from the second light-shielding region, the black level is determined from the output of the pixel in the second light-shielding region, and the image of the effective region is determined based on the black level. Image correction means for correcting
An imaging device comprising: When there is a saturated region where the output is saturated in the effective region,
The image correction unit excludes the pixels in the second light-shielding region on the same horizontal line as the saturation region from the second light-shielding region, and determines the black level from the output of the pixels in the second light-shielding region. The image pickup apparatus according to claim 1, wherein In the first light-shielding region, the image correction unit reads the output of the image sensor from the top to the bottom or reads from the bottom to the top, and the defective pixel information is read from the pixel in the matching region from which the pixel output is read. 3. The image pickup apparatus according to claim 1, wherein the defective pixel registered in the above is excluded, and the black level is determined from the output of the pixel in the first light shielding region. The image pickup apparatus according to claim 1, wherein the first light-shielding region is provided on an upper end portion of the image pickup element. The image pickup apparatus according to any one of claims 1 to 4, wherein the second light-shielding region is provided at an end portion on either the left side or the right side of the image pickup element. Having multiple image sensors,
A connecting means for connecting the image data picked up by the plurality of image pickup devices,
The image pickup apparatus according to claim 1, wherein a celestial sphere image is generated from the two pieces of image data that have been joined together. The image pickup apparatus according to claim 6, wherein the image data picked up by the plurality of image pickup elements have different read directions. A first light-shielding region having a pixel whose output is not read by reading the output of the pixel from top to bottom or from bottom to top;
Whether the pixel output is read from top to bottom or from bottom to top, a second light-shielding region having a pixel whose output is read, and
An image pickup device having an effective area for outputting image data for incident light;
An image correction method performed by an image pickup apparatus having a storage unit for storing defective pixel information in which the coordinates of defective pixels are registered, and having an image pickup device whose pixel output is read from top to bottom or from bottom to top. ,
The image correction means excludes the defective pixel registered in the defective pixel information from the second light-shielding region, determines a black level from the output of the pixel of the second light-shielding region, and based on the black level. Correct the image of the effective area,
An image correction method characterized by the above. A first light-shielding region having a pixel whose output is not read by reading the output of the pixel from top to bottom or from bottom to top;
Whether the pixel output is read from top to bottom or from bottom to top, a second light-shielding region having a pixel whose output is read, and
An image pickup device having an effective area for outputting image data for incident light;
A defective pixel information storage unit in which the coordinates of the defective pixel are registered; and an image pickup device having an image pickup device whose pixel output is read from top to bottom or from bottom to top,
The defective pixel registered in the defective pixel information is excluded from the second light-shielding region, the black level is determined from the output of the pixel in the second light-shielding region, and the image of the effective region is determined based on the black level. Image correction means for correcting
Program to function as.

Images