JP2012119959A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus which improves the filling rate of pixels by optical shift in rearrangement of the pixels in a high resolution space, thereby improving the effect of super-resolution processing.SOLUTION: The imaging apparatus comprises: an imaging element 31 which comprises plural pixels, photoelectrically converts light from a subject, and outputs a pixel signal; a lens unit 42 which guides the light from the subject to the imaging element; and an optical shift mechanism 35 which causes relative circular motion between an optical image formed on a light-receiving surface of the imaging element and the imaging element. The radius of the circular motion is set to be not smaller than a pixel pitch in the imaging element.

Description

  The present invention performs high-resolution processing by super-resolution processing from a plurality of original images acquired by so-called pixel shift, which performs imaging while relatively displacing the optical image formed on the light receiving surface of the imaging device and the imaging device. The present invention relates to an imaging apparatus suitable for generating an image.

  The imaging device employs a two-dimensional image sensor in which pixels are arranged in a matrix. In this two-dimensional image sensor, the resolution depends on the size and the number of pixels. On the other hand, it is higher than the original resolution of the image sensor from a plurality of original images acquired by so-called pixel shifting, which is performed while relatively displacing the optical image formed on the light receiving surface of the image sensor and the image sensor. A technique for generating a resolution image is conventionally known.

  In such a technique for increasing the resolution by shifting pixels, a shift mechanism (hereinafter referred to as “optical shift mechanism”) that relatively displaces the optical image and the image sensor is referred to as “optical shift”. For example, a technique for minutely displacing the image sensor with an actuator made of a piezo element is known (see Patent Document 1).

  In addition, a parallel plate is disposed between the image pickup optical system and the image pickup device so as to be inclined with respect to the optical axis of the image pickup optical system, and the parallel plate is rotated around the optical axis so as to be on the light receiving surface of the image pickup device. There is a technique for shifting the position of an optical image (see Patent Document 2).

  Further, as an image processing method for generating a high-resolution image from a plurality of original images acquired by pixel shifting, an image shift process for mapping pixel values of a low-resolution image to pixels of a high-resolution image, or an ML (Maximum-likelihood) method A super-resolution technique using a MAP (Maximum A Posterior) method, a POCS (Projection On to Convex Sets) method, or the like is known (see Patent Document 1).

JP 2008-306492 A JP 2000-125170 A

  By the way, in the super-resolution processing technology, the number of low-resolution original images used for processing is increased, and the quality of the high-resolution image (substantial resolution) obtained by using many original images with slightly different imaging positions. Can be improved. And the low-resolution original image used for a super-resolution process can be acquired with the technique which rotates the parallel plate disclosed by the said patent document 2. FIG. In Patent Document 2, the pixel relative to the optical image is shifted so that the pixel of the imaging element is shifted by 1/2 of the pixel pitch in the vertical and horizontal directions with respect to the optical image formed on the light receiving surface of the imaging element. By setting a radius of a circular motion (hereinafter referred to as “shift radius”), the pixel density can be substantially doubled.

  However, for example, when the super-resolution processing is applied to a single-plate color image sensor in which RGB pixels are arranged in a Bayer arrangement, the super-resolution processing is performed with a shift amount of ½ of the pixel pitch as in Patent Document 2. There is a problem that it is difficult to obtain a high-resolution image without sufficiently exhibiting the above effect. Therefore, as a result of intensive studies by the inventor of the present application, the pixel center of the image sensor (relatively viewed in other words, that is, an optical image formed on the image sensor) moves in a high-resolution space by optical shift. The trajectory when performing the processing is low in the ratio (hereinafter referred to as “pixel filling rate”) that fills (passes) each pixel area (sub-pixel area) in the high-resolution space, and high resolution even when super-resolution processing is performed. It was found that the pixels were not sufficiently rearranged in the space (see FIGS. 11 and 12 described later).

  The present invention has been devised in view of such problems of the prior art, and by improving the pixel filling rate by optical shift when rearranged in a high resolution space, super-resolution processing is achieved. An object of the present invention is to provide an image pickup apparatus that can improve the above effect.

  An imaging apparatus according to the present invention includes a plurality of pixels, an image sensor that photoelectrically converts light from a subject and outputs a pixel signal, an optical system that guides light from the subject to the image sensor, and the image sensor And an optical shift mechanism for performing a relative circular motion between the optical image formed on the light receiving surface and the image sensor, and the radius of the circular motion is greater than or equal to the pixel pitch in the image sensor. To do.

  As described above, according to the present invention, the radius of the circular motion by the optical shift mechanism is set to be equal to or larger than the pixel pitch in the image sensor, and the pixel filling rate by the optical shift when rearranged in the high resolution space is improved. As a result, it is possible to improve the super-resolution processing effect.

1 is an overall configuration diagram of a network camera system according to a first embodiment of the present invention. FIG. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus and an image processing apparatus illustrated in FIG. Schematic diagram showing the processing status in the imaging apparatus and image processing apparatus shown in FIG. Sectional drawing which shows the imaging part of the imaging device shown in FIG. 4 is an exploded perspective view of the optical shift mechanism shown in FIG. FIG. 4 is a perspective view of the optical shift mechanism shown in FIG. FIG. 4 is a plan view of the optical shift mechanism shown in FIG. FIG. 4 is a cross-sectional view of the main part of the optical shift mechanism shown in FIG. The block diagram of the shift control part 14 in the optical shift mechanism shown in FIG. Sectional drawing which shows the incident condition of the light to the image pick-up element shown in FIG. Schematic diagram showing the situation of the relative circular motion (r = 0.5L) of the pixel with respect to the light image 11 is an enlarged view of the main part of FIG. Schematic diagram showing the situation of the relative circular motion (r = 0.9L) of the pixel with respect to the optical image The schematic diagram which shows the condition of the relative circular motion (r = 1.0L) of the pixel with respect to a light image The schematic diagram which shows the condition of the relative circular motion (r = about 1.35L) of the pixel with respect to a light image The schematic diagram which shows the condition of the relative circular motion (r = 2L) of the pixel with respect to a light image The graph which shows the relationship between the shift radius r and the filling factor Fu of a pixel in an imaging device A graph showing the relationship between the shift radius r and the amount of movement of the pixel center due to optical shift Schematic diagram showing the state of imaging and the image generated by this Schematic diagram showing the situation of the imaging reference position in an example of the ratio between the imaging period and the circular motion period

  A first invention made to solve the above problems includes an image sensor that has a plurality of pixels, photoelectrically converts light from a subject and outputs a pixel signal, and outputs light from the subject to the image sensor. An optical system for guiding, and an optical shift mechanism for performing a relative circular motion between the optical image formed on the light receiving surface of the image sensor and the image sensor, the radius of the circular motion being the image sensor The pixel pitch in FIG.

  According to this, by setting the radius of circular motion by the optical shift mechanism to be greater than or equal to the pixel pitch in the image sensor and improving the pixel filling rate due to optical shift when rearranged in a high resolution space, The effect of image processing can be improved.

Moreover, 2nd invention sets it as the structure where the radius of the said circular motion is defined by the following formula | equation.
((0.5L) ² + (1.25L) ² ) ^1/2 ≦ r ≦ 2L
Here, r is the radius of the circular motion, and L is the pixel pitch.

  According to this, the pixel filling rate due to the optical shift when rearranged in the high resolution space can be set to a high value, and the effect of the super-resolution processing can be further improved.

  According to a third aspect of the invention, the radius of the circular motion is twice as large as the pixel pitch.

  According to this, the pixel filling rate due to optical shift when rearranged in the high resolution space can be maximized, and the effect of the super-resolution processing can be surely exhibited.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the term “axial direction” indicates the optical axis direction (corresponding to the vertical direction in FIG. 4), and the term “radial direction” is the direction orthogonal to the optical axis (corresponding to the horizontal direction in FIG. 4). It shall be shown. The radial direction can take any angle within 360 ° around the optical axis.

  A fourth invention includes a single-plate color image sensor having a plurality of pixels, photoelectrically converting light from a subject and outputting a pixel signal, and an optical system for guiding light from the subject to the image sensor. And an optical shift mechanism for performing a relative circular motion between the optical image formed on the light receiving surface of the image sensor and the image sensor, and a locus caused by the circular motion of the G pixel of the color image sensor. Is a circle formed by the optical shift mechanism so as to pass 14/16 or more of a plurality of sub-pixel regions to be generated by increasing the resolution with respect to a region of interest corresponding to one pixel size of the color imaging device. The radius of motion is set.

  According to this, the pixel filling rate in the super-resolution processing is sufficient, and a high-quality high-resolution image can be obtained.

(First embodiment)
FIG. 1 is an overall configuration diagram of a network camera system according to a first embodiment of the present invention. As shown in FIG. 1, a network camera system to which the present invention is applied includes at least one imaging device (here, a network camera) 1 and an image processing device 2. The imaging device 1 and the image processing device 2 are connected via the Internet 3, and imaging data generated by the imaging device 1 is transmitted to, for example, the image processing device 2 that exists in a remote place, and a video is captured by the image processing device 2. Is displayed. Various command signals for controlling the imaging device 1 are transmitted from the image processing device 2 to the imaging device 1.

  The imaging data is transmitted from the imaging apparatus to the image processing apparatus using a so-called Internet protocol such as TCP (UDP) / IP. For example, the imaging data is encrypted and encapsulated, for example, in a VPN (Vertual Private Network). ) Or a network camera system called CCTV (Closed Circuit TV) in which the imaging device 1 and the image processing device 2 are connected in a one-to-one relationship by a dedicated line. .

  FIG. 2 is a block diagram illustrating a schematic configuration of the imaging apparatus and the image processing apparatus illustrated in FIG. 1. As shown in FIG. 2, the imaging device 1 includes an imaging unit 11, an image processing unit 12, a data compression / transmission unit 13, and a shift control unit 14. The imaging unit 11 includes an imaging element 31 that photoelectrically converts light from a subject and outputs an analog pixel signal. The image sensor 31 is a two-dimensional CMOS image sensor. The image sensor 31 has, for example, a quad VGA (1280 × 960 pixels) pixel configuration, a sensor size of 1/3 inch, and a pixel pitch L that is an interval between adjacent pixels is 3.75 μm in both the main scanning direction and the sub-scanning direction. It is. Instead of this, a two-dimensional CCD image sensor may be used for the image sensor 31.

  The analog signal output from the image sensor 31 is converted into a digital signal by the A / D converter 32, and this digital signal is input to the image processing unit 12, where color correction, demosaic processing, gradation correction (γ correction). , YC separation processing and the like are performed and converted into image data. This image data is stored in the data compression / transmission unit 13 by, for example, H.264. The image data is transmitted to the image processing apparatus 2 after being subjected to compression processing such as H.264 or MPEG4.

  As will be described in detail later, the imaging unit 11 includes an optical shift mechanism 35 that relatively slightly displaces the optical image formed on the light receiving surface of the imaging element 31 and the imaging element 31. In the optical shift mechanism 35, an optical member (see reference numeral 51 in FIG. 4) that minutely displaces the optical image is rotated by a magnetic force, and the radial and axial positions of the optical member are respectively set. A position control unit 38 for controlling, a radial magnetic sensor 39 and an axial magnetic sensor 40 for detecting the radial and axial positions of the optical member, respectively, are provided.

  The optical shift mechanism 35 is controlled by the shift control unit 14. The rotating portion of the optical shift mechanism 35 is provided with a magnetized portion (not shown) here, and both magnetic sensors 39 and 40 detect the position information of the magnetized portion and output it to the shift control portion 14. The shift control unit 14 controls the magnetic rotation driving unit 36 based on this position information to rotate the optical member, and controls the position control unit 38 to hold the optical member at a predetermined position.

  The image processing apparatus 2 includes a data reception decoding unit 21, a display unit 22, a storage unit 23, a super-resolution processing unit 24, a period setting unit 25, and an input unit 26. The image processing apparatus 2 is configured by installing necessary application software in an information processing apparatus such as a personal computer or a workstation, or may be a dedicated apparatus such as a CCTV recorder.

  In the image processing device 2, the compressed image data transmitted from the imaging device 1 is received and decoded by the data receiving / decoding unit 21, converted into RGB image data, and displayed in real time by a display or the like. 22 is displayed. Further, the RGB image data is sent to the storage unit 23 composed of a hard disk drive device or the like, temporarily accumulated therein, and can be read from the storage unit 23 and reproduced on the display unit 22 as necessary.

  For example, when a high-resolution image is necessary for verification of traffic accidents, etc., the image data is read from the storage unit 23 and super-resolution processing is performed by the super-resolution processing unit 24 so that the high-resolution image is obtained. (Still image) can be generated and the high resolution image can be displayed on the display unit 22.

  As will be described in detail later, the input unit 26 receives an input of an imaging cycle from the user and sends it to the cycle setting unit 25. The cycle setting unit 25 determines a circular motion cycle based on the imaging cycle sent from the input unit 26 and transmits a command signal related to the circular motion cycle to the imaging device 1. The shift control unit 14 of the imaging apparatus 1 drives the optical member at a rotational speed corresponding to the designated circular motion cycle by operating the optical shift mechanism 35 based on the command signal related to the circular motion cycle.

  Further, the cycle setting unit 25 transmits information related to the imaging timing to the drive circuit 33, whereby a timing generator (not shown) of the drive circuit 33 generates a synchronization pulse for control at a predetermined interval. This synchronization pulse is output to the image sensor 31, thereby determining the start and end of the charge accumulation period (electronic shutter period) of the image sensor 31. Further, the synchronization pulse is also output to the shift control unit 14, whereby the shift control unit 14 can grasp the period during which the image sensor 31 is actually exposed.

  FIG. 3 is a schematic diagram illustrating processing states in the imaging device 1 and the image processing device 2. As shown in FIG. 3, the image sensor 31 is driven by a drive circuit 33, and imaging (sampling) is performed at a constant cycle (hereinafter, imaging cycle) according to a timing signal generated by the drive circuit 33. For example, when generating 30 frame images per second at a frame rate of 30 frames / sec, the imaging cycle is set to about 30 ms.

  The super-resolution processing unit 24 of the image processing apparatus 2 performs super-resolution processing for generating a high-resolution image from a plurality of temporally continuous frame images. In this super-resolution processing, first, the frame image stored in the storage unit 23 is displayed as a still image by frame advance. Then, when the reference image is designated by the user from among them, the frame image serving as the reference image and a plurality of frame images before and after the frame image are read from the storage unit 23 and sent to the super-resolution processing unit 24 for super Resolution processing is performed.

  As super-resolution processing, for example, ML (Maximum-likelihood) method, MAP (Maximum A Posterior) method, POCS (Projection On to Convex Sets) method, etc. are adopted and realized by executing application software on CPU. Is done. In general, since the super-resolution processing has a large amount of calculation, a part of the processing may be performed using a GPU (Graphics Processing Unit) or dedicated hardware.

  Here, the ML method uses a square error between a pixel value of a low-resolution image estimated from a high-resolution image and an actually observed pixel value as an evaluation function, and a high resolution that minimizes this evaluation function. This is a method of using an image as an estimated image. That is, the ML method is a super-resolution processing method based on the principle of maximum likelihood estimation. The MAP method is a method for estimating a high-resolution image that minimizes an evaluation function obtained by adding probability information of a high-resolution image to a square error. That is, the MAP method is a super-resolution processing method that estimates a high-resolution image as an optimization problem that maximizes the posterior probability by using some foresight information for the high-resolution image. The POCS method is a super-resolution processing method for obtaining a high-resolution image by creating simultaneous equations regarding pixel values of a high-resolution image and a low-resolution image and sequentially solving the equations.

  In these super-resolution processes, first, a high-resolution image is assumed, and based on the point spread function (PSF function) obtained from the assumed high-resolution image from the camera model, the pixel value of the pixel of the low-resolution image is determined. And a process of searching for a high-resolution image that reduces the difference between the estimated value and the observed pixel value (observed value). Therefore, these super-resolution processes are called reconfigurable super-resolution processes.

  Here, the process of searching for a high-resolution image is a process called so-called “positioning” that searches for a position corresponding to a pixel obtained as a low-resolution image in the high-resolution image. In general, in super-resolution processing, even if the change in pixel position between a plurality of low-resolution images is unknown, it is possible to increase the resolution, so that the alignment processing is repeatedly performed over a wide range around the pixel of interest. For this reason, it is known that the calculation cost becomes extremely high. On the other hand, as described in detail later, in the present invention, the position of the pixel shifted by the optical shift mechanism 35 is known, and each frame image, that is, a low-resolution image is captured at the known position. For at least a stationary subject, much of the alignment processing can be omitted by optical shift, and the calculation cost can be greatly reduced.

  Note that super-resolution processing that uses image information that spans a plurality of temporally continuous frames may be particularly referred to as inter-frame reconstruction type super-resolution processing. On the other hand, when the reconstruction type super-resolution processing is performed within one frame, it is called intra-frame reconstruction type super-resolution. In this embodiment, inter-frame reconstruction type super-resolution is adopted.

  Note that here, the still image that has been increased in resolution by super-resolution processing is reproduced in the image processing device 2, but if the processing capability of the image processing device 2 is sufficiently high, it can be obtained by super-resolution processing. It is also possible to reproduce a moving image using a high-resolution image as a frame image.

  4 is a cross-sectional view showing the imaging unit 11 of the imaging device 1 shown in FIG. 2, and FIG. 5 is an exploded perspective view of the optical shift mechanism 35 of the imaging device 1 shown in FIG. FIG. 7 is a perspective view of the optical shift mechanism 35, FIG. 7 is a plan view of the optical shift mechanism 35, and FIG. 8 is a cross-sectional view of the main part of the optical shift mechanism. In FIG. 5, for convenience of explanation, each member (optical member 51 and the like) accommodated inside the optical capsule (capsule member) 50 is shown in a form taken out to the outside. 5 and 6, the rotational drive coil 81 shown in FIG. 7 is omitted.

  As illustrated in FIG. 4, the imaging unit 11 of the imaging device 1 forms an image on the light receiving surface 31 a of the imaging device 31 with light from a sensor module 41 provided with the imaging device 31 and a subject (not shown). A lens unit (optical system) 42 and an optical shift mechanism 35 that displaces a light image formed on the light receiving surface 31 a of the image sensor 31 in a direction perpendicular to the optical axis C are provided. The lens unit 42 is supported on the substrate 46 via the lens holder 45. The sensor module 41 and the optical shift mechanism 35 are also supported by the substrate 46. Note that other electrical components and the like are mounted on the substrate 46 as necessary.

  The optical shift mechanism 35 mainly includes, for example, an optical capsule 50 formed of a transparent resin or a glass material, and an optical member 51 accommodated in the optical capsule 50. Inside the optical capsule 50, a liquid (fluid) 52 having a refractive index higher than that of air is sealed. Further, the refractive index of the liquid 52 is set to be smaller than the refractive index of a parallel plate 53 described later.

  The optical member 51 has a substantially disk shape, and a parallel plate 53 inclined at a predetermined angle with respect to the optical axis C of the lens unit 42 is provided at the center thereof. The parallel plate 53 can be formed from a material such as optical glass or acrylic resin. Further, on the outer peripheral side of the optical member 51, an annular back yoke 54 that supports the outer peripheral portion, an annular radial magnet 55 attached to the outer peripheral portion of the back yoke 54, and an optical member in the back yoke 54 There are provided an upper magnet (first shaft side magnet) 56 and a lower magnet (second shaft side magnet) 57 respectively fixed to the light incident side and light emitting side portions 51 (here, the upper and lower surfaces of the back yoke 54). It has been. Both the upper magnet 56 and the lower magnet 57 have an annular shape with the same diameter. These members 54 to 57 are arranged coaxially with the optical axis C as the center, and are accommodated together with the optical member 51 in the optical capsule 50 filled with the liquid 52.

  The optical capsule 50 is composed of two members 50a and 50b that are overlapped in the vertical direction, whereby a sealed space in which the liquid 52 is enclosed is formed. The optical capsule 50 has a constricted cross section at the center. The optical capsule 50 is connected to the central portion 60 that defines a disk-shaped space with the optical axis C as the center, and to the outer peripheral side of the central portion 60 and has a rectangular cross section. And an annular portion 61 that defines an annular space. The central portion 60 accommodates the parallel flat plate 53, while the annular portion 61 widened in the vertical direction accommodates the outer peripheral portion of the optical member 51, the back yoke 54, and the like. Here, the distance between the inner surface of the optical capsule 50 and the surface of the optical member 51 is set to 0.5 mm. In the optical capsule 50, at least the optical path portion around the optical axis C in the central portion 60 (the region through which incident light from the lens unit 42 passes) may be formed of the transparent material as described above.

  Here, the distance D between the lens unit 42 in which the optical capsule 50 (the central portion 60) is arranged and the sensor module 41 is usually very narrow (for example, about 3 mm), and the photographing range becomes a wide angle. It becomes narrower accordingly. However, if only the optical member 51 (parallel plate 53) is arranged in the optical path, the apparatus can be configured without any problem even though the distance D is narrow.

  Further, the upper surface 60a of the central portion 60 of the optical capsule 50 is in contact with the lower surface 42a of the lens unit 42, whereby the optical path between the optical capsule 50 and the lens unit 42 (strictly, incident light passes). Space) is sealed. Further, the lower surface 61 a of the annular portion 61 is in contact with the upper surface 46 a of the substrate 46, so that the optical path between the optical capsule 50 and the sensor module 41 (that is, the optical capsule 50, the sensor module 41, and the substrate 46). The defined space V) is sealed. With such a configuration, it is possible to prevent deterioration in image quality due to intrusion of dust or the like into the optical path from the lens unit 42 to the image sensor 31. As a method for sealing the optical path as described above, the optical capsule 50 may be brought into direct contact with the image sensor as long as the imaging element 31 is regarded as a molded image sensor itself.

  Note that portions other than the portion corresponding to the optical path of the optical capsule 50 may be opaque (for example, black). By doing in this way, what is called stray light into which unnecessary light enters into the image sensor 31 can be blocked.

  In this embodiment, an antifreeze liquid (a mixture of polypropylene glycol and water) is used as the liquid 52. When the liquid 52 contains water, the back yoke 54 formed of a magnetic material (for example, an iron-based material) may be rusted. In order to prevent this, a resin coating is applied to the surface of the back yoke 54. Note that the antifreeze liquid does not need to be aqueous, and for example, transparent silicone oil may be employed. In this case, there is no risk of rusting on the diameter side magnet 55 and the like, and rust prevention treatment such as resin coating is not required. Moreover, when the installation environment of the imaging device 1 is limited to a room, it is not always necessary to use antifreeze. Although the magnetic characteristics are slightly inferior, the formation of rust can be suppressed by forming the back yoke 54 from a nonmagnetic metal material such as SUS316 or a resin.

  As shown in FIG. 5, in the optical shift mechanism 35, the optical member 51 and the radial magnet 55 are not in contact with other than the fluid by applying a magnetic force to the radial magnet 55 around the optical capsule 50. The magnetic member 51 is rotated in a non-contact manner with the inner wall of the optical capsule 50, and the magnetic force is applied to the radial magnet 55, the upper magnet 56, and the lower magnet 57. And three position control units 38 that hold the axial position at a regular position (a fixed position where the parallel plate 53 can rotate without contacting the optical capsule 50) As shown in FIG. 7, it is called position control part 38a, 38b, 38c.). As described above, the diameter side magnet 55, the upper magnet 56, and the lower magnet 57 of the optical member 51 are all permanent magnets, and an electromagnet is employed in the position control unit 38 that applies a magnetic force to them. No signal line is drawn into the capsule 50, and the sealing property of the optical capsule 50 can be improved.

  As will be described in detail later, the rotational drive and radial position control of the optical member 51 are performed by three radial magnetic sensors 39 (hereinafter, when it is necessary to distinguish each of them) as the position detector shown in FIG. As shown in FIG. 7, this is performed based on information output from the radial magnetic sensors 39a, 39b, and 39c. Further, the axial position control of the optical member 51 is performed based on information output by the axial magnetic sensor 40 as the position detection unit shown in FIG.

  With such a configuration, the optical member 51 rotates in a non-contact state within the optical capsule 50 filled with the liquid 52. The configuration using this magnetic force is similar to the configuration of a so-called bearingless motor, there is no sliding portion, the optical shift mechanism 35 is driven with extremely low vibration, and a long life can be achieved. it can.

  As shown in FIG. 6, an origin sensor 65 that periodically detects the origin position accompanying the rotation of the optical member 51 is provided around the optical capsule 50. The origin sensor 65 is composed of a reflection type photosensor (photoreflector), and detects a marker 66 (see FIG. 4) provided on the upper side of the diameter side magnet 55. The origin sensor 65 is not limited to a reflective photosensor, and a known sensor including another optical sensor can be used. The parallel flat plate 53 inclined with respect to the optical axis C is detected by the magnetic pole position of the magnetized portion 68 (see FIG. 7) of the radial magnet 55 detected by the output of the origin sensor 65 and the output of the radial magnetic sensor 39. It is possible to reliably grasp the light shift direction due to.

  As shown in FIG. 4, the marker 66 is disposed so as to face the origin sensor 65 with the optical capsule 50 interposed therebetween. The marker 66 is optically identified. For example, the marker 66 may be printed in a color different from that of the magnet on the radial side magnet 55, or the surface roughness of the magnet may be changed to result in a different reflectance. You may be made to obtain. The origin sensor 65 detects the marker 66 using the difference in reflectance between the marker 66 and the diameter side magnet 55.

  In the rotation direction of the optical member 51, the relationship between the inclination direction of the parallel plate 53 and the position of the marker 66 in the rotation direction is grasped in advance, so that the direction of the parallel plate 53 (that is, optical) at the timing when the marker 66 is detected. The direction of the target shift). That is, since the rotation angle of the parallel plate 53 can be known when the marker 66 is detected, the direction in which the optical shift is performed can be specified. Here, the optical shift amount by the parallel plate 53 depends on the angle of the parallel plate 53 with respect to the optical axis, but this angle is detected by the axial magnetic sensor 40 and is made constant by the magnetic circuit shown in FIG. Be controlled. Further, since the rotating body including the back yoke 54 is rotated, the shaft shake is suppressed to be small due to the gyro effect similarly to the rotation of the frame, so that the angle of the parallel plate 53 with respect to the optical axis (that is, the optical shift amount) is approximately. It becomes constant. Therefore, the coordinates of the optical image shift due to the optical shift can be calculated from the rotation angle of the parallel plate 53.

  By the way, in the super-resolution processing performed by the super-resolution processing unit 24 shown in FIG. 2, the imaging position is on the order of submicron (for example, the pixel pitch L is 5.6 μm, and this is set to 4 by super-resolution processing. In the case of enlargement to x4, the newly generated pixel pitch L is 5.6 / 4 = 1.4 μm, and displacement of sub-micron order is necessary). It is known that the calculation cost can be significantly reduced. In the imaging apparatus and network camera system according to the present invention, the origin sensor 65 as described above is employed, and the light when the imaging device 31 performs imaging in accordance with the timing signal (synchronization pulse) generated by the drive circuit 33. Since the image shift position is determined, the calculation cost of the super-resolution processing can be reduced. That is, according to the present invention, since the shift position of the image can be reliably grasped, the search amount of the above-described high-resolution image is reduced, and the calculation cost of the super-resolution processing for the still portion of the imaging target of the optical shift is reduced. Is greatly reduced.

  As shown in FIG. 7, the radial magnet 55 includes a magnetized portion 68 including 16 magnetic poles alternately magnetized to the N and S poles along the circumferential direction. The diameter-side magnet 55 is a so-called plastic magnet formed of polyphenylene sulfide resin (PPS) in which minute magnetic particles are dispersed and mixed, thereby preventing water absorption and swelling even in the liquid 52 containing water. Is done. Here, neodymium is adopted as the magnetic particles. A neodymium magnet having an extremely large magnetic force can obtain a large driving torque, and thus is effective when the viscosity of the liquid 52 becomes large at low temperatures. On the other hand, since neodymium can be oxidized by water and cause rust, the surface of the diameter-side magnet 55 is coated with a resin material to prevent contact with the liquid 52. Similarly, the upper magnet 56 and the lower magnet 57 are made of PPS in which neodymium magnetic particles are dispersed and mixed, and these are also coated with a resin. The magnetic particles used as the diameter-side magnet 55, the upper magnet 56, and the lower magnet 57 are not limited to neodymium, and for example, ferrite, samarium cobalt, or the like can also be used.

  The magnetic rotation drive unit 36 includes a stator core 80 formed by stacking a plurality of electromagnetic steel strips, and a rotation drive coil 81 formed of a conductive wire wound around the stator core 80. The substantially U-shaped stator core 80 is disposed at equal intervals along the outer peripheral surface in a state where both ends are in contact with the outer wall of the optical capsule 50, and the diameter-side magnet 55 is interposed via the optical capsule 50. Opposed to the magnetized portion 68. Thereby, the optical shift mechanism 35 has the same configuration as a so-called inner rotor type three-phase motor (12 slots, 16 poles in this embodiment). The magnetic rotation driving unit 36 rotates the magnetizing unit 68 in a predetermined direction by an attractive force and a repulsive force acting on the magnetizing unit 68 by a magnetic field generated by passing a current through the rotation driving coil 81. As a result, the back yoke 54 and the optical member 51 (parallel plate 53) rotate around the optical axis C.

  The radial magnetic sensors 39a, 39b, and 39c are composed of Hall elements, and are arranged at equal intervals in the circumferential direction along the outer peripheral surface in a state where the radial magnetic sensors 39a, 39b, and 39c are in contact with the outer wall of the optical capsule 50. It is located between the two magnetic rotation driving parts 36. With such arrangement of the radial magnetic sensors 39a, 39b, 39c, the radial displacement amount of the optical member 51 can be accurately detected. By using a Hall element as the radial magnetic sensor 39, the position of the optical member 51 in the optical capsule 50 can be detected with an accuracy of about 1 μm. If higher accuracy is required, an optical position sensor (PSD: Position Sensitive Detector) may be used instead of the Hall element. By using PSD, the position detection accuracy is improved to about 0.1 μm.

  Note that there is no contradiction between the point that the detection accuracy when the Hall element is used as the position detection means is 1 μm and the above-described “determine the imaging position in submicron order”. In the optical shift mechanism 35 of this embodiment, the optical member 51 (parallel plate 53) has only power for shifting incident light in parallel, and the shift position of the optical image is determined solely by the rotation angle of the optical member 51. The Even if the optical member 51 is displaced parallel to the optical axis direction or the radial direction, the shift position by the parallel plate 53 is not affected at all, and if it is affected, the inclination angle of the optical member 51 with respect to the optical axis C is not affected. This is a case where fluctuates. In this embodiment, as will be described in detail later, the influence of the tilt angle variation is reduced, and as a result, even if the detection accuracy is about 1 μm, the shift position is determined with submicron order accuracy.

  In the configuration of the 12-slot 16-pole shown in FIG. 7, six spaces are generated between the adjacent magnetic rotation driving units 36 on the outer periphery of the optical capsule 50. Therefore, the three radial magnetic sensors 39a, 39b, 39c can be arranged in these spaces at positions rotated by 180 degrees around the three position control units 38a, 38b, 38c and the optical axis C, respectively. Accordingly, it is possible to easily control the position of the optical member 51 in the radial direction while effectively reducing the size of the apparatus by effectively using the space outside the optical capsule 50.

  The radial magnetic sensor 39 detects the magnetic force generated from the magnetized portion 68 of the radial magnet 55, thereby outputting position information of the magnetic poles (N pole and S pole) of the magnetized portion 68. More specifically, when one set of the N pole and the S pole of the radial magnet 55 moves relative to the radial magnetic sensor 39 relative to itself, the shift control unit 14 uses a sine wave having this as one cycle as position information. (See FIG. 2). The shift control unit 14 calculates the rotation speed of the optical member 51 by processing this phase information. That is, the radial magnetic sensor 39 and the shift control unit 14 function as a rotation position detection unit and a rotation speed detection unit of the optical member 51.

  Instead of such a method of detecting the rotational speed, for example, marking is performed on the outer peripheral portion (magnetized portion or the like) of the optical member 51 or the diameter-side magnet 55, and this is detected by an optical sensor (photo reflector). May be. In this case, if the black-and-white marking is used, the rotation angular velocity can be detected at a higher sampling rate because drawing can be performed at a relatively narrow pitch.

  The axial magnetic sensor 40 is disposed in the vicinity of the position control unit 38 in contact with the upper outer wall of the optical capsule 50. The axial magnetic sensor 40 is disposed so as to face the upper magnet 56 via the optical capsule 50 as shown by a two-dot chain line in FIG. The axial magnetic sensor 40 may be disposed on the front side. The axial magnetic sensor 40 detects the magnitude of the magnetic force generated from the upper magnet 56, and outputs axial position information of the back yoke 54 (ie, the optical member 51) to which the upper magnet 56 is attached. Based on this position information, the shift control unit 14 (see FIG. 2) controls the current flowing through the axial position control coil 71 (see FIG. 8), and controls the position of the optical member 51 in the axial direction.

  As shown in FIG. 8, the position control unit 38 protrudes upward (in a direction parallel to the optical axis direction) at the lower end portion of the stator core 70 having a substantially E shape in which a plurality of electromagnetic steel strips are stacked. An axial position control coil 71 wound around the protrusion 70a and a radial position control coil 72 wound around a protrusion 70b protruding laterally (in a direction perpendicular to the optical axis direction) at the intermediate portion of the stator core 70. And have. The tips of the protrusions 70a and 70b are in contact with the lower surface and the outer peripheral surface of the optical capsule 50, respectively.

  The electromagnet 74 formed on the protrusion 70 a by the axial position control coil 71 functions as an axial magnetic force generation unit that generates a magnetic force acting on the lower magnet 57 and is used for axial position control of the optical member 51. It is done. In addition, the electromagnet 75 formed on the protrusion 70 b by the radial position control coil 72 functions as a radial magnetic force generation unit that generates a magnetic force acting on the radial magnet 55, and the radial position of the optical member 51. Used for control.

  In addition, a protrusion 70 c that protrudes downward (in a direction parallel to the optical axis direction) is provided at the upper end of the stator core 70 so as to face the protrusion 70 a with the optical capsule 50 interposed therebetween. The protrusion 70 c is formed as a part of the permanent magnet 76, and its tip is in contact with the upper surface of the optical capsule 50. The permanent magnet 76 functions as an axial magnetic force generating means for generating a magnetic force acting on the upper magnet 56, cooperates with the lower electromagnet 74 facing thereto, and is used for axial position control of the optical member 51. In this case, the permanent magnet 76 and the upper magnet 56 have the same magnetic pole (here, both N poles), and a repulsive force acts between them. In addition, although the neodymium magnet is employ | adopted as the permanent magnet 76, considering the balance with the magnetic force by the lower electromagnet 74, you may use another magnet (for example, a ferrite magnet etc.).

  The protrusion 70 b is formed as a part of a connecting member 77 that connects the upper and lower portions of the stator core 70. The connecting member 77 is made of a non-magnetic material (for example, resin, ceramic, etc.) in order to prevent magnetic interaction between the electromagnet 74, the electromagnet 75, and the permanent magnet 76.

  In the present embodiment, the position control unit 38 functions as means for controlling the radial position and the axial position of the optical member 51. However, the position control unit 38 controls the radial position and the axial position. It is also possible to employ a configuration in which the position control units are individually provided.

  In the axial position control, the electromagnet 74 is excited by passing a current through the axial position control coil 71, and a repulsive force is generated between the electromagnet 74 having the same magnetic pole (here, the S pole) and the lower magnet 57. . The position of the optical member 51 in the axial direction is balanced by balancing the magnitude of this repulsive force with the repulsive force acting between the permanent magnet 76 located on the opposite side of the electromagnet 74 and the upper magnet 56 across the optical capsule 50. Is held in the normal position.

  On the other hand, when the electromagnet 74 is not excited, the repulsive force from the permanent magnet 76 acts on the optical member 51, the optical member 51 moves downward, and the attractive force between the lower magnet 57 and the non-energized stator core ( That is, the lower magnet 57 is fixed in a state where the lower end of the lower magnet 57 is in contact with the optical capsule 50. As described above, when the image pickup apparatus is not turned on (for example, when the image pickup apparatus is transported or the like), the optical member 51 is securely fixed, so that the parallel plate 53 can be prevented from being damaged. . In this case, it is preferable that the optical member 51 (parallel flat plate 53) does not contact the optical capsule 50 from the viewpoint of preventing damage even in the fixed state. Therefore, when the optical member 51 is in the normal position shown in FIG. 8, the distance d1 between the lower magnet 57 and the optical capsule 50 is set to be smaller than the distance d2 between the optical member 51 and the optical capsule 50. The

  Note that the optical shift mechanism 35 may be configured such that an attractive force acts between the permanent magnet 76 and the upper magnet 56 and between the electromagnet 74 and the lower magnet 57 to balance them. However, by applying a repulsive force as described above, a frictional force is generated in the optical member 51 due to the influence of the magnetic force (attraction force) by the upper magnet 56 or the lower magnet 57 at the time of initial startup or the like, and the apparatus cannot be started. There is an advantage that troubles can be prevented.

  In the radial position control, the electromagnet 75 is excited by passing a current through the radial position control coil 72, and a repulsive force is generated between the radial magnet 55. This repulsive force acts from three directions around the radial side magnet 55 due to the arrangement of the position control units 38a, 38b, and 38c shown in FIG. 7, and by balancing them, the radial position of the optical member 51 is fixed. Retained.

  In the optical shift mechanism 35, as shown in FIG. 7, the position control unit 38a and the radial magnetic sensor 39a, the position control unit 38b and the radial magnetic sensor 39b, and the position control unit 38c and the radial magnetic sensor 39c are Are arranged at positions rotated by 180 ° around the optical axis C. The radial position control coils 72 (see FIG. 8) of the position controllers 38a, 38b, and 38c are driven based on the position information detected by the radial magnetic sensors 39a, 39b, and 39c, respectively. That is, the radial position control coils 72 of the placement control units 38a, 38b, and 38c are driven based on position information detected at positions facing each other across the optical capsule 50, respectively. With such a configuration, the increase in the output of the radial magnetic sensor 39 is directly related to the increase in the separation distance between the corresponding position control unit 38 and the optical member 51 (diameter side magnet 55), and the position control process is performed. Becomes easy.

  Further, in the optical shift mechanism 35, as shown in FIGS. 6 and 7, the magnetic rotation driving unit 36, the radial magnetic sensor 39, the axial magnetic sensor 40, and the position control unit 38 are each provided in the optical capsule 50. It is made to contact with the outer surface. In this way, the components necessary for the rotational drive control and position control of the optical member 51 are arranged in contact with the optical capsule 50. Thereby, if the dimensional accuracy of the optical capsule 50 itself is managed well, the positional relationship (that is, the distance relationship) of each component can be determined with extremely high accuracy, and high control performance can be realized. Here, as a preferred example, all the components necessary for the rotation drive control and the position control are in contact with the optical capsule 50. However, the present invention is not limited to this, and some of these components contact the optical capsule 50. A configuration that touches is also possible.

  Further, in the optical shift mechanism 35, an example in which a configuration corresponding to a so-called three-phase motor is applied to rotate the optical member 51 is shown, but the same applies to a configuration similar to a single-phase motor. An effect can be obtained.

  FIG. 9 is a configuration diagram of the shift control unit 14 in the optical shift mechanism 35 shown in FIG. Hereinafter, the rotational drive control and the position control of the optical member 51 by the shift control unit 14 will be described with reference to FIGS. 9 and 7 together.

  First, the rotational drive control of the optical member 51 will be described. As shown in FIG. 7, the radial magnet 55 of the optical member 51 is provided with a 16-pole magnetized portion 68, and the magnetism generated by the magnetized portion 68 is detected by three radial magnetic sensors 39. Is done. The detection output (analog signal) of the radial magnetic sensor 39 is input to the comparator 91a. Before the input, the signal level is changed to ± with respect to the ground level by an amplifier and an analog offset circuit (not shown). It is shifted like analog. The comparator 91a outputs an FG signal (pulse signal) when detecting a zero cross of the analog signal. This FG signal is input to the arithmetic processing unit 92 as an interrupt signal (IRQ). As described above, since there are three radial magnetic sensors 39, the arithmetic control unit 92 receives an interrupt signal based on zero cross for each sensor. The detection output of the origin sensor 65 is binarized by the comparator (CMP) 91b and output to the arithmetic processing unit 92 as an interrupt signal (IRQ). Further, the synchronization pulse output from the drive circuit 33 is input as an interrupt signal (IRQ) to the arithmetic processing unit 92 of the shift control unit 14.

  Here, since the interval between the magnetic poles of the magnetized portion 68 is set to be the same, the distance between the magnetic poles is known. Therefore, the speed calculation value Vn can be obtained by dividing this distance by the time between pulses counted by the high-speed counter provided in the shift control unit 14 (for example, the interval between adjacent zero crosses). That is, in this embodiment, the arithmetic processing unit 92 also functions as a speed calculation unit, and calculates the rotation speed of the optical member 51 based on the output of the radial magnetic sensor 39. With such a configuration, it is possible to detect the speed and position of the optical member 51 with the radial magnetic sensor 39, and the cost of the apparatus can be reduced.

Here, details of the processing in the arithmetic processing unit 92 will be described. The arithmetic processing unit 92 includes a CPU 93 that executes arithmetic processing, a ROM 94 that stores a program necessary for the processing of the CPU 93, and a RAM 95 that functions as a work area when the CPU 93 executes the program. The arithmetic processing unit 92 rotates the optical member 51 at a constant angular speed by performing PI control (proportional integral control) based on the calculated rotation speed of the optical member 51. Since the interval (time) of the FG pulse when the optical member 51 rotates at a constant speed is known, this is the speed theoretical value Vr (set angular speed, that is, speed target value), and the speed relative to the speed theoretical value Vr. The error δV of the calculated value Vn is calculated as follows:
δV = Vr−Vn

The speed theoretical value Vr and the speed calculated value Vn are originally real numbers, but can be normalized in advance as, for example, a 16-bit integer value in order to speed up the calculation by the CPU 93. The P term, which is a proportional term in the PI control, is calculated as follows by multiplying δV by an appropriate gain Gp.
P = Gp × δV

Here, if the gain is infinite, speed control can be performed using only the P term. Generally, the gain is finite, and at this time, a speed offset is generated. Therefore, the error δV is integrated and appropriate. The integral gain I is multiplied to calculate an integral term I (Integral) term as in the following equation.
I = Gi × Σ (δV)

  This P + I is a so-called speed command value, and the CPU 93 sends the speed command value to the pulse width modulator (PWM) 96. Based on this speed command value, the pulse width modulator 96 calculates an ON duty ratio in a predetermined period, and a signal (PWM signal) modulated based on the duty ratio is output to the three-phase driver 97. The three-phase driver 97 includes three systems of push-pull type transistor circuits 98 inside, and controls the current flowing through each rotation drive coil 81 that is star-connected based on the PWM signal output from the shift control unit 14 to rotate. Drive. Here, the common signal extracted from each rotary drive coil 81 is input to the three-phase driver 97, and the phase excitation (ON / OFF) of each push-pull transistor circuit 98 is controlled. The rotation of the optical shift mechanism is controlled by switching the phase excitation. For this current control, a known voltage regulator may be used instead of the pulse width modulator 96. Whichever configuration is employed, the optical member 51 can be rotated at a plurality of different constant angular velocities by changing the speed target value used as the speed theoretical value Vr. As a result, the rotational drive control is executed in a circular motion cycle instructed from the cycle setting unit 25 (see FIG. 2).

  Next, position control of the optical member 51 will be described. As described above, the position detection of the optical member 51 is performed by the three axial magnetic sensors 40 and the three radial magnetic sensors 39 that also serve as input sources of FG signals (speed information). These sensor groups detect the magnetism of the magnetized portions provided in the upper magnet 56 and the diameter-side magnet 55, respectively. This detected value is converted into a digital signal by the A / D converters 101a and 101b. The outputs of the A / D converters 101a and 101b are 8 bits, and the CPU 93 of the arithmetic processing unit 92 receives a total of 8 × 3 × 2 bits of position information signals.

  Then, the CPU 93 calculates a deviation from a theoretical value (positional relationship (known value) between the radial magnetic sensor 39 and the axial magnetic sensor 40 when the optical member 51 is in the normal position) based on the acquired position information signal. A correction value to be corrected is calculated, and this correction value (digital signal) is sent to the D / A converters 102a and 102b. In the D / A converters 102a and 102b, the digital signal is converted into an analog signal. Based on this analog signal, the radial drive driver 103 and the axial drive driver 104 drive the radial position control coil 72 and the axial position control coil 71, respectively, to position the optical member 51 in the optical capsule 50. Control.

  In the present embodiment, the outputs of the radial magnetic sensor 39 and the axial magnetic sensor 40 directly indicate distance information. Therefore, calculation only needs to be performed for the I term in the PI control.

  FIG. 10 is a cross-sectional view showing a state of incidence of light on the image sensor. FIG. 10A shows a state where the optical path of the incident light is shifted to the rightmost side, and FIG. 10B shows the state of FIG. ) Shows a state in which the parallel flat plate 53 is rotated 180 °. When the parallel flat plate is further rotated 180 ° from the state shown in FIG. 10B, the state returns to the state shown in FIG.

  As shown in FIG. 10, since the parallel flat plate 53 of the optical member 51 is inclined with respect to the optical axis C of the lens unit 42, the light incident through the lens unit 42 is refracted and the light receiving surface of the image sensor 31. When the position of the light incident on the optical plate 51 changes according to the rotational position of the parallel plate 53 and the optical member 51 is rotated by the optical shift mechanism 35, a light image formed on the light receiving surface of the image sensor 31 becomes an optical member. It moves so as to draw a circle with a period (circular motion period) corresponding to the rotational speed of 51, and thereby the optical image can be relatively displaced relative to the image sensor 31.

  Next, the relative circular motion of the pixel with respect to the optical image in the imaging apparatus having the above configuration will be described with reference to FIGS. In the following, an example in which the shift radius r is changed stepwise in the optical shift mechanism of the imaging apparatus will be described. However, for the sake of convenience, a value outside the scope of the present invention (that is, the value of the shift radius r that does not provide the effect of the present invention) ). Here, the shift radius r is expressed in terms of the pixel pitch L of the image sensor (that is, as a multiple of the pixel pitch L). For example, when the shift radius r is set to 5.6 μm corresponding to the pixel pitch L, it is simplified as “r = 1L” as necessary.

  The shift radius r can be changed by adjusting the thickness and inclination angle of the optical path portion in the parallel plate 53 and the polypyrene glycol concentration. For example, when r = 1L, the thickness of the parallel flat plate 53 is 0.1 mm, the inclination angle is 20 °, and the polyglycol glycol concentration is 60 wt% (the mixing ratio of PG and water is 6: 4. The refractive index is about 1.38).

  Further, as shown in FIG. 10, the relative circular motion of the pixel with respect to the optical image is such that the optical image is displaced with respect to the pixel of the fixed image pickup device 31. The relative movement of the light image is illustrated so that the pixels of the image sensor 31 move with respect to the stationary light image. Further, each pixel has a light receiving area of a predetermined size for receiving light. However, in the following description, only the center position (pixel center) of each pixel is illustrated for convenience.

  FIG. 11 is a schematic diagram illustrating a state of relative circular motion of a pixel with respect to an optical image in the case of r = 0.5L, and FIG. 12 is an enlarged view of a main part of FIG. Note that r = 0.5L corresponds to the shift amount of Patent Document 2 described above.

  As illustrated in FIG. 11, the image sensor 31 includes an R pixel that receives an R (Red) component, a B pixel that receives a B (Blue) component, and a G pixel that receives a G (Green) component. Each pixel is a so-called single-plate color imaging device in which the pixels are arranged based on a so-called Bayer arrangement. In this Bayer array, G pixels are arranged in a zigzag pattern (checker flag shape) with the number of pixels being 1/2 of the total number of pixels, and R pixels and B pixels are G with a number of pixels that is 1/4 of the total number of pixels. The pixels are dispersedly arranged at positions other than the pixel arrangement position.

  In the figure, the X axis indicates the main scanning direction, and the Y axis indicates the sub scanning direction. In addition, a plurality of straight lines in the X-axis and Y-axis directions indicated by solid lines in the drawing are arranged at intervals of the pixel pitch L of the image sensor 31. Further, a plurality of straight lines in the X-axis and Y-axis directions indicated by broken lines in the drawing are for indicating pixels in the high resolution space in the super-resolution processing described above. Here, pixels in the high resolution space Is defined as a subpixel obtained by dividing the pixel pitch L into four for both main scanning and subscanning (that is, when super-resolution processing of 4 × 4 times is performed). The same applies to FIGS. 12 to 16.

  In this embodiment, description will be made on the premise of super-resolution processing for increasing the resolution of a low-resolution image to 4 × 4 times. However, the advantageous effect obtained by 4 × 4 times is smaller than this. Even in the case (for example, the pixel pitch L is divided into three, that is, super-resolution of 3 × 3 times), it can be obtained in the same manner.

  As indicated by a solid circle in FIG. 11, the pixel center of each G pixel moves while drawing a circular orbit whose radius is ½ of the pixel pitch L with each black circle (●) as the rotation center. Here, in order to determine the movement range of each G pixel, a determination region indicated by a one-dot chain line in FIG. 12 (that is, 16 subpixels between the rotation centers of two predetermined G pixels adjacent to each other). Pay attention to the configured area). In this determination area, six sub-pixels (hereinafter referred to as “effective sub-pixels”) filled with diagonal lines are the movement range of the pixel center of the G pixel (the range in which the pixel can be scanned by circular motion). Moreover, pixel information is lost in the high-resolution space in the super-resolution processing for the 10 sub-pixels that are not filled with other diagonal lines.

  Since the G component has the most influence on the resolution in terms of human visual characteristics, the effect of the super-resolution processing for improving the resolution is directly improved by improving the G pixel filling rate in the high resolution space. The

  When r = 0.5L, the pixel filling factor Fu (here, the ratio of effective sub-pixels in 16 sub-pixels) is as small as 6/16 (37.5%). It is difficult to demonstrate. For example, even when the shift radius r is slightly increased to r = (√2 / 2) L (about 0.71 L), the pixel filling rate Fu is the same (Fu = 6/16).

  FIG. 13 is a schematic diagram illustrating a state of the relative circular motion of the pixel with respect to the optical image when r = 0.9L. Note that FIG. 13 (the same applies to FIGS. 14 to 16) corresponds to FIG. 12, and matters not particularly mentioned below are the same as those in FIGS. 11 and 12 described above. When r = 0.9L, the number of effective sub-pixels is 10, and the pixel filling factor Fu is increased to 10/16 (62.5%) than when r = 0.5L. However, since the pixel filling rate is still low, it is difficult to exert the effect of super-resolution processing.

  When the filling rate is 10/16 or less, granular noise is generated particularly in the edge portion of the image with high resolution by the super-resolution processing, and the super-resolution effect is limited. Further, this granular noise tends to be accompanied by a color that should not exist in the object to be photographed, that is, a pseudo color. As for the degree of pseudo color, the lower the filling rate, the more prominent the primary color component, and the R component and B component are often the case. The reason seems to be that the filling rate of the R pixel and the B pixel, which originally have a small number of pixels, is lower than the filling rate of the G pixel.

  FIG. 14 is a schematic diagram illustrating a state of a relative circular motion of a pixel with respect to an optical image when r = 1.0L. In the case of r = 1.0L, the number of effective subpixels is 12, and the pixel filling rate Fu is 12/16 (75%). In this case, the pixel filling rate Fu is significantly improved as compared with the case where r = 0.5L. For example, even when the shift radius r is further increased to r = 1.25L, the pixel filling rate Fu is the same as in the case of r = 1.0L (Fu = 1/16).

  When the filling rate is about 12/16, the granular noise and false color at the edge portion are reduced, and the point of sensory evaluation is greatly improved. However, these granular noises and false colors remain, and are not sufficient for high-resolution image reproducibility.

FIG. 15 is a schematic diagram showing a state of relative circular motion of a pixel with respect to an optical image in the case of r = ((0.5 L) ² + (1.25 L) ² ) ^1/2 (about 1.35 L). is there. The locus (shift locus) of the circular motion at this time is, for example, S1, and the center of rotation is “●” and the radius “r” is determined by the point “▲”. In the case of r = ((0.5L) ² + (1.25L) ² ) ^1/2 , there are 14 effective subpixels, and the pixel filling factor Fu is 14/16 (87.5%), which is high. The value pixel filling factor Fu can be realized. For example, even when the shift radius r is further increased to r = √2L (about 1.41L) or r = 1.9L, the pixel filling rate Fu is the same (Fu = 14/16).

  FIG. 16 is a schematic diagram illustrating a state of the relative circular motion of the pixel with respect to the optical image in the case of r = 2L. In the case of r = 2L, all 16 subpixels become effective subpixels, and the pixel filling rate Fu can be set to the maximum 16/16 (100%).

  When the filling rate is 14/16 or more, granular noise and pseudo color at the edge portion are not generated at all. For this reason, it is desirable that the trajectory when performing the optical shift on the premise of the super-resolution processing is not limited to the case where the trajectory is a circle, but is designed so that the filling rate is 14/16 or more.

  As described above, in the present embodiment, the trajectory due to the circular motion of a plurality of green pixels of the single-plate color image sensor (in the case of FIGS. 15 and 16, the circular motion of six green pixels is involved in pixel filling). However, 14/16 or more of the plurality of sub-pixel areas to be generated by increasing the resolution for the target area corresponding to one pixel size of the color image sensor (the “determination area” described with reference to FIG. 11). It can also be said that the radius of circular motion by the optical shift mechanism is set so as to pass.

FIG. 17 is a graph showing the relationship between the shift radius r and the pixel filling factor Fu in the imaging apparatus. As shown in FIG. 17, the pixel filling factor Fu shown on the vertical axis tends to generally increase as the shift radius r shown on the horizontal axis increases when 0.5L ≦ r <2L. When r = 2L, the pixel filling rate Fu is maximum (16/16). Thereafter, the pixel filling rate Fu repeatedly increases and decreases between 14/16 and 16/16 until r = 3L. . For example, when r = ((0.5L) ² + (1.25L) ² ) ^1/2 (about 2.12L), Fu = 14/16, and when r = 2.25L, Fu = When r = 2.5L, Fu = 14/16, and when r = 3.0L, Fu = 16/16.

As described above, in the imaging device of the present application, Fu ≧ 12/16 (75%) when at least 1L ≦ r, the effect of super-resolution processing can be improved. In the case of ((0.5L) ² + (1.25L) ² ) ^1/2 ≦ r, since Fu ≧ 14/16 (87.5%), the effect of high super-resolution processing is achieved. This makes it possible to produce high-resolution images. In the case of r = 2, Fu = 16/16 (100%), which is particularly preferable, and an extremely high super-resolution effect can be exhibited.

  Here, even when the shift radius r exceeds 2, the pixel filling factor Fu may be 14/16 (87.5%) or 16/16 (100%). When the rotation speed is constant (angular speed), the rotation speed (that is, the displacement speed of the optical image) increases due to the increase of the shift radius r. ) High-frequency components are lost due to pixel integration, which is a factor that suppresses the effect of super-resolution processing. Therefore, it is desirable that the shift radius r is smaller in consideration of the occurrence of blurring and the like. Therefore, the shift radius r in consideration of the charge accumulation period of the image sensor and the influence of blur will be described next.

  FIG. 18 is a graph showing the relationship between the shift radius r and the amount of movement of the pixel center due to optical shift. Here, a plurality of charge accumulation periods (5 ms to 30 ms) are shown. Further, the rotational speed of the relative circular motion of the pixel with respect to the optical image is set to 60 rpm.

In FIG. 18, the movement amount M shown on the vertical axis is the movement amount of the pixel due to the optical shift in each charge accumulation period, and here, the movement amount M is shown on the basis of the size of the subpixel shown in FIG. (Hereinafter, the subpixel size is expressed as _LSUB .) That is, M = 1L _SUB indicates that the pixel has moved by the size L _{SUB of the} sub pixel. Here, when the movement amount M exceeds 1L _SUB (that is, the pixel movement range extends to a plurality of subpixels), information on a plurality of subpixels that should be originally separated is added (integrated). The aliasing distortion increases and the effect of the super-resolution processing decreases. On the other hand, when the movement amount M is 1L _SUB or less, the information of individual subpixels is not lost, and blurring hardly occurs in the super-resolution processing. Therefore, the shift radius r is preferably selected to be a value that increases the filling factor Fu in a range where the movement amount M is 1L _SUB or less.

For example, in the case of performing the 4 × 4 super-resolution processing as described above, when the charge accumulation period is 15 ms and the shift radius r is 2.75 or less, the movement amount M is 1 L _SUB or less. Accordingly, in consideration of the pixel filling factor Fu in order to obtain the effect of the super-resolution processing, the shift radius r is ((0.5L) ² + (1.25L) ² ) ^1/2 ≦ r ≦ 2.75. Set within the range. Further, if the pixel filling rate Fu is the same, the shift radius r is desirably smaller. Therefore, the shift radius r is preferably ((0.5L) ² + (1.25L) ² ) ^1/2 ≦ r. It is good to set in the range of ≦ 2. Furthermore, since the pixel filling factor Fu is 16/16 (100%), the shift radius r is more preferably set to r = 2.

  Next, imaging (sampling) will be described. FIG. 19 is a schematic diagram illustrating imaging and a situation of an image generated by imaging. Here, as shown in FIG. 19, frame images F1 and F2 in which the image is captured while the relative circular motion of the pixel with respect to the optical image is continuously performed in one direction at a constant speed, and the imaging positions are slightly shifted. Are sequentially generated. The imaging reference positions P1, P2,... Shown indicate the timing of imaging, and one frame image is generated for each. In particular, here, the center position of the pixel at the start of imaging is shown as the imaging reference position, charge accumulation is started at each imaging reference position, and charge accumulation is completed immediately before the imaging reference position, and a pixel signal is output. Is done.

  The rotational speed of this circular motion is stably maintained by the above-described PI control, the reference of the rotational position of the parallel plate 53 (see FIG. 10 etc.) is managed by the above-described origin sensor 65 (see FIG. 6), and the parallel plate Since the influence of the inclination angle variation of 53 on the shift width (shift position) is kept small, the imaging position of the frame image captured at each timing can be grasped with extremely high accuracy.

  Hereinafter, description will be made with reference to FIGS.

  Information regarding the imaging position is sequentially generated by the arithmetic processing unit 92 based on the outputs of the radial magnetic sensor 39 and the origin sensor 65 (both refer to FIG. 9), and the shift control unit 14 including the arithmetic processing unit 92 transmits to the storage unit 23. The image data is output and stored in association with image data in units of frames output from the imaging unit 11. In the course of the super-resolution processing, information regarding the imaging position is referred to by the super-resolution processing unit 24, and the alignment processing is simplified at this time (both refer to FIG. 2).

  Note that in order to achieve an appropriate high resolution by super-resolution processing, it is desirable that all pixels be in a uniformly displaced state, and it is not appropriate to cause a time difference in charge accumulation timing between pixel lines. Here, a global shutter system is adopted in which the shutter operation of all the pixels is performed at the same timing.

  In addition, by performing many imaging (sampling) while the pixel performs one circular motion, the quality of the high resolution image obtained by the super-resolution processing can be improved. Set to a non-integer multiple of the imaging period. In this way, by repeating the circular motion, it is possible to capture images at a number of different positions, so a large number of images with slightly different imaging positions can be generated, so the high resolution obtained by super-resolution processing The image quality can be improved. On the other hand, if the circular motion cycle is an integral multiple of the imaging cycle, the imaging reference position does not change even if the circular motion is repeated, and the number of imaging reference positions that can be set by one circular motion is limited.

  Hereinafter, an example of the imaging reference position will be described with reference to FIG. 20 by specifically determining the ratio between the circular motion period and the imaging period. FIG. 20 is a schematic diagram illustrating the state of the imaging reference position in an example of the ratio between the imaging cycle and the circular motion cycle. In FIG. 20, the pixel pitch is shown as 1.

  In the example shown in FIG. 20, the circular motion cycle is set to 7.5 times the imaging cycle. Here, if the imaging cycle is 30 ms (about 30 frames / s), for example, the circular motion cycle is 225 ms (= 30 ms × 7.5). In this case, the imaging reference position returns to the original position at the second round motion, and 15 times of imaging (sampling) is performed while the circular motion is performed twice. Each imaging reference position is separated by a relative angle of 48 deg (= 360 deg / 7.5). Note that since the super-resolution effect is suppressed when there is blurring (integration effect) in the image, the selection of the charge accumulation period (shutter speed) in the image sensor 31 is important. From this point of view, the shutter speed should be as high as possible. However, on the other hand, if the shutter speed is increased, sensitivity is likely to be insufficient. Therefore, it is desirable to compensate the gain with priority on the shutter speed according to the amount of light of the subject. In this embodiment, the shutter speed is set to 1/250 sec (4 ms). When the shutter speed was set to this level for the imaging cycle = 30 ms, no adverse effect due to image blurring in the super-resolution processing was observed visually.

  In the first round movement, as shown in FIG. 20A, imaging is performed at the imaging reference positions P1 to P8, and in the second round movement, as shown in FIG. 20B, the imaging reference position P9 is taken. The imaging reference positions P9 to P15 are intermediate positions between the adjacent imaging reference positions (for example, P1 and P2) in the first round motion. When the first and second circular motions are combined, the imaging reference positions P1 to P15 are separated from each other by a relative angle of 24 degrees as shown in FIG.

  Here, the first processing mode in which super-resolution processing is performed based on eight images obtained by imaging at the first imaging reference positions P1 to P8 of the circular motion, and the first and second times of the circular motion. And two processing modes, the second processing mode for performing the super-resolution processing, based on the 15 images obtained by imaging at the imaging reference positions P1 to P15.

  In the first processing mode, two imaging reference positions having different positions in both the X-axis and Y-axis directions are set within the original one pixel range, so that the image sensor in each of the X-axis and Y-axis directions. The resolution can be increased at a resolution almost twice the original resolution of 31. On the other hand, in the second processing mode, four imaging reference positions having different positions in both the X-axis and Y-axis directions are set within the original one pixel range. High resolution can be achieved with a resolution almost four times the original resolution of the image sensor 31.

  In particular, in this second processing mode, each of the imaging reference positions P9 to P15 set at the second circular motion is adjacent to the imaging reference positions P1 to P8 set at the first circular motion. Since the imaging reference position is uniformly distributed without being biased, an image excellent in compatibility with the super-resolution processing can be generated.

  It is also possible to perform super-resolution processing in the imaging device 1 while imaging is being performed in the imaging device 1. In this case, in the second processing mode, circular motion is performed twice and 15 What is necessary is just to perform a super-resolution process once whenever the image of a sheet is prepared.

  On the other hand, in the first processing mode, it is preferable to perform super-resolution processing each time eight images are aligned while sequentially shifting the imaging reference position. Specifically, super-resolution processing is performed using the eight images obtained by imaging at the imaging reference positions P1 to P8 at the first time, and at the imaging reference positions P9 to P15 and P1 at the second time. Super-resolution processing is performed using eight images obtained by imaging, and thereafter, imaging reference positions P2 to P9 for the third time, imaging reference positions P10 to P15, P1, and P2 for the fourth time. Are shifted one by one.

  In this way, two processing modes can be set. In both modes, it is not necessary to change the circular motion cycle (the rotational speed of the optical shift mechanism 35) or the imaging cycle, and thus control is easy.

  Note that the first image used for the super-resolution processing in each mode does not need to be limited to the image obtained by imaging at the original position P1, and in the first processing mode, one circular motion is performed from an arbitrary position. Super-resolution processing is performed using 8 images captured during the execution, and in the second processing mode, 15 images captured during two circular motions are performed from an arbitrary position. It may be used to perform super-resolution processing.

  Such processing is performed when super-resolution processing is performed using frame images stored in the storage unit 23 of the image processing device 2 as shown in FIG. The present invention can also be applied to the case where super-resolution processing is performed, and particularly in the latter case, it is not necessary to return the imaging start position to the original position P1 when the processing mode is switched. It is possible to generate high-resolution images with different resolutions by switching.

  As shown in FIG. 2, the imaging cycle is designated by the user using the input unit 26 in the image processing apparatus 2, and the circular motion cycle is determined based on the designated imaging cycle by the cycle setting unit 25. A command signal related to the circular motion period determined here is transmitted to the imaging apparatus 1. The shift control unit 14 of the imaging device 1 operates the optical shift mechanism 35 at a rotational speed corresponding to the designated circular motion cycle based on the command signal regarding the circular motion cycle acquired from the image processing device 2.

  Further, the user can designate the processing mode (first processing mode and second processing mode), and uses the frame image stored in the storage unit 23 of the image processing apparatus 2 as shown in FIG. When performing super-resolution processing, the user is allowed to specify the processing mode together with the reference image, and the number of frame images corresponding to the processing mode specified here is set based on the frame image specified as the reference image. It is only necessary to read out and perform super-resolution processing.

  In addition, the said circular motion period can be changed suitably. For example, by setting the circular motion cycle to 7.2 times the imaging cycle, the imaging reference position returns to the original position at the fifth circular motion, and 36 imagings (sampling) while the circular motion is performed five times. It is also possible to adopt a configuration in which In this case, the imaging reference positions are separated by a relative angle of 50 deg (= 360 deg / 7.2).

  Although this invention was demonstrated based on specific embodiment containing an Example, these embodiment is an illustration to the last and this invention is not limited by these embodiment. Note that all the components of the imaging apparatus according to the present invention shown in the above-described embodiments are not necessarily essential, and can be appropriately selected as long as they do not depart from the scope of the present invention.

  The image pickup apparatus according to the present invention can improve the effect of super-resolution processing by improving the pixel filling rate due to optical shift when rearranged in a high-resolution space, and can receive light from the image pickup device. Suitable for generating a high-resolution image by super-resolution processing from multiple original images acquired by so-called pixel shift, which performs imaging while relatively displacing the optical image formed on the surface and the image sensor. It is useful as an imaging device.

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Image processing apparatus 11 Imaging part 12 Image processing part 14 Shift control part 31 Imaging element 35 Optical shift mechanism 36 Magnetic rotation drive part 38 Position control part 39 Radial direction magnetic sensor 40 Axial direction magnetic sensor 41 Sensor module 42 Lens Unit (optical system)
50 Optical capsule 51 Optical member 52 Liquid (fluid)
53 Parallel plate 54 Back yoke 55 Diameter side magnet 56 Upper magnet 57 Lower magnet 65 Origin sensor 72 Position control coil 75 Electromagnet 76 Permanent magnet 77 Connecting member C Optical axis

Claims (4)

An imaging device having a plurality of pixels, photoelectrically converting light from a subject and outputting a pixel signal;
An optical system for guiding light from the subject to the image sensor;
An optical shift mechanism for performing a relative circular motion between the optical image formed on the light receiving surface of the image sensor and the image sensor;
The radius of the said circular motion is more than the pixel pitch in the said image pick-up element, The imaging device characterized by the above-mentioned. The imaging apparatus according to claim 1, wherein the radius of the circular motion is determined by the following expression.
((0.5L) ² + (1.25L) ² ) ^1/2 ≦ r ≦ 2L
Here, r is the radius of the circular motion, and L is the pixel pitch.   The imaging apparatus according to claim 2, wherein a radius of the circular motion is twice as large as the pixel pitch. A single-plate color image sensor having a plurality of pixels, photoelectrically converting light from a subject and outputting a pixel signal;
An optical system for guiding light from the subject to the image sensor;
An optical shift mechanism for performing a relative circular motion between the optical image formed on the light receiving surface of the image sensor and the image sensor;
The trajectory due to the circular motion of the G pixel of the color image sensor is 14/16 or more of the plurality of sub-pixel areas to be generated by increasing the resolution with respect to the region of interest corresponding to one pixel size of the color image sensor. An imaging apparatus, wherein a radius of circular motion by the optical shift mechanism is set so as to pass through

Images