JP2012129613A - Imaging system, image processing apparatus for use in the same, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus for generating a high resolution image from a plurality of low resolution images different in imaging location which implements a reduced computational cost by suppressing superfluous registration processing in super resolution processing.SOLUTION: An image processing apparatus 2 includes: a shift cancellation section 102 for canceling a shift of a low resolution image on the basis of imaging location to generate a shift-canceled image; a motion detection section 103 for comparing the shift-canceled image with a reference image to detect a motion in the low resolution image; a registration processing section 106 for registering the shift-canceled image with the reference image to generate a registration image; and a reconstruction processing section 107 for generating a high resolution image by reconstruction processing. The reconstruction processing section is configured to process a given region to be processed by using the registration image if any motion has been detected or by using the low resolution image or shift-canceled image if no motion has been detected.

Description

  The present invention performs super-resolution processing from a plurality of low-resolution images acquired by so-called pixel shifting, which performs imaging while relatively displacing the optical image formed on the light receiving surface of the imaging element and the imaging element. The present invention relates to an imaging system suitable for generating a resolution image, an image processing apparatus used therefor, an image processing method, and an image processing program.

  An imaging apparatus in this type of imaging system employs a two-dimensional image sensor in which pixels are arranged in a matrix. In this two-dimensional image sensor, the resolution depends on the size of the pixel and the number of pixels. On the other hand, it is higher than the original resolution of the image sensor from a plurality of original images acquired by so-called pixel shifting, which is performed while relatively displacing the optical image formed on the light receiving surface of the image sensor and the image sensor. A technique for generating a resolution image is conventionally known.

  In such a technique for increasing the resolution by shifting pixels, a shift mechanism (hereinafter referred to as “optical shift mechanism”) that relatively displaces the optical image and the image sensor is referred to as “optical shift”. For example, a technique for minutely displacing the image sensor with an actuator made of a piezo element is known (see Patent Document 1).

  In addition, a parallel plate is disposed between the image pickup optical system and the image pickup device so as to be inclined with respect to the optical axis of the image pickup optical system, and the parallel plate is rotated around the optical axis so as to be on the light receiving surface of the image pickup device. There is a technique for shifting the position of an optical image (see Patent Document 2).

  Further, as an image processing method for generating a high-resolution image from a plurality of original images acquired by pixel shifting, an image shift process for mapping pixel values of a low-resolution image to pixels of a high-resolution image, or an ML (Maximum-likelihood) method A super-resolution technique using a MAP (Maximum A Posterior) method, a POCS (Projection On to Convex Sets) method, or the like is known (see Patent Document 1). In super-resolution processing technology, the number of low-resolution original images used for processing is increased, and the quality (substantial resolution) of the resulting high-resolution images is improved by using many original images with slightly different imaging positions. Can be made.

JP 2008-306492 A JP 2000-125170 A

  By the way, in the above-mentioned patent document 1, a stationary object such as a screen of a liquid crystal display panel is used as a subject. However, when the imaging apparatus is applied to a monitoring camera or the like, a subject (for example, a person or a car) is used. May move. In general super-resolution processing, using the fact that the imaging position changes irregularly due to camera shake, etc., the reference image and other frame images from multiple frame images obtained by imaging are highly accurate in a high-resolution space. A so-called “positioning process” is performed to match the two.

  However, in such an alignment process, it is necessary to repeat numerical operations for each pixel in the high-resolution space, and thus enormous operations are required. In addition, in the registration processing for the captured image of the conventional imaging apparatus, the movement of the known subject caused by the optical shift as described above is not distinguished from the movement that is inherently unpredictable caused by the subject itself moving. As a result, there is a problem that a lot of useless calculation processing occurs and the calculation cost increases.

  The present invention has been devised in view of the above-described problems of the prior art, and an imaging system capable of reducing calculation costs by suppressing useless alignment processing in super-resolution processing. An object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program used therefor.

  The image pickup apparatus of the present invention acquires a plurality of low-resolution images picked up by causing a relative circular motion between the image pickup element and a light image formed on the light receiving surface of the image pickup element, and these image pickup positions. An image processing apparatus for generating a high resolution image from a plurality of low resolution images having different positions, and based on information on an imaging position of the low resolution image, the position of the low resolution image is set as a reference low resolution image position. Based on a result of the motion detection, a shift cancel unit that generates a matched shift cancel image, a motion detection unit that detects a motion in the low resolution image by comparing the shift cancel image and the reference low resolution image, and An alignment processing unit that generates an alignment image by aligning the shift cancel image with the reference low-resolution image, and a reconstruction process. A reconstruction processing unit that generates a high-resolution image, and the reconstruction processing unit detects the movement of the predetermined processing target area of the shift cancel image when the motion detection unit detects motion. A configuration in which the reconstruction process is executed using an image, and on the other hand, if no motion is detected by the motion detection unit, the reconstruction process is executed using the low resolution image or the shift cancel image; To do.

  As described above, according to the present invention, the shift cancel image is generated in the low resolution space based on the information of the imaging position, and the motion detection unit does not detect the low resolution image serving as the motion reference and the shift cancel image. In all cases, the low-resolution image or the shift-cancelled image whose imaging position is already known is used to perform reconstruction processing in the high-resolution space, so that unnecessary registration processing in super-resolution processing is suppressed. It has an excellent effect that the calculation cost can be reduced.

1 is an overall configuration diagram of a network camera system according to a first embodiment of the present invention. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus and an image processing apparatus according to a first embodiment. Schematic diagram showing the processing status in the imaging apparatus and the image processing apparatus according to the first embodiment Sectional drawing which shows the imaging part of the imaging device which concerns on 1st Embodiment. 1 is an exploded perspective view of an optical shift mechanism according to a first embodiment. The perspective view of the optical shift mechanism which concerns on 1st Embodiment. The top view of the optical shift mechanism concerning a 1st embodiment Sectional drawing of the principal part of the optical shift mechanism which concerns on 1st Embodiment. Sectional drawing which shows the incident condition of the light to the image pick-up element which concerns on 1st Embodiment The block diagram of the shift control part 14 in the optical shift mechanism which concerns on 1st Embodiment. The figure which shows the detection result of the marker 66 and the magnetized part 68 by the origin sensor 65 and the radial direction magnetic sensor 39 which concern on 1st Embodiment. Explanatory drawing of the position detection method by the shift control part 14 which concerns on 1st Embodiment. The schematic diagram which shows the condition of the relative circular motion of the pixel with respect to the optical image which concerns on 1st Embodiment. The schematic diagram which shows the condition of the imaging reference position in an example of the ratio of the imaging period and circular motion period which concern on 1st Embodiment. Functional block diagram of the super-resolution processing unit 24 in the image processing apparatus 2 according to the first embodiment. Flow chart showing a flow of super-resolution processing by the super-resolution processing unit 24 according to the first embodiment. Explanatory drawing which shows an example of the process of the captured image in the super-resolution process which concerns on 1st Embodiment The block diagram which shows the process of each frame image in the super-resolution process which concerns on 1st Embodiment Functional block diagram of the super-resolution processing unit 24 according to the second embodiment. Explanatory drawing which shows an example of the process of the captured image in the super-resolution process which concerns on 2nd Embodiment Explanatory drawing which shows the position shift of the imaging position which arises in the relative circular motion of the pixel with respect to the optical image which concerns on 3rd Embodiment. Explanatory drawing which shows the position shift of the imaging position which arises in the relative circular motion of the pixel with respect to the optical image which concerns on 3rd Embodiment. Functional block diagram of the super-resolution processing unit 24 according to the third embodiment. Flow chart showing the flow of super-resolution processing by the super-resolution processing unit 24 in the third embodiment.

  A first invention made to solve the above-described problems is a plurality of low-resolution images captured by causing relative circular motion between an image sensor and a light image formed on the light-receiving surface of the image sensor. An image processing apparatus that acquires an image and generates a high-resolution image from a plurality of low-resolution images having different imaging positions, and based on the information on the imaging position of the low-resolution image, the position of the low-resolution image as a reference A shift cancel unit that generates a shift cancel image that matches the position of the reference low resolution image, and a motion detection unit that detects motion in the low resolution image by comparing the shift cancel image and the reference low resolution image; , Based on the result of the motion detection, aligning the shift cancel image with the reference low resolution image to generate an alignment image. A processing unit, and a reconstruction processing unit that generates a high-resolution image by reconstruction processing, wherein the reconstruction processing unit detects motion in the predetermined processing target region of the shift cancel image by the motion detection unit. In this case, the reconstruction process is executed using the alignment image. On the other hand, if no motion is detected by the motion detection unit, the reconstruction process is performed using the low resolution image or the shift cancel image. The configuration processing is executed.

  According to this, when a shift cancel image is generated in the low resolution space based on the information of the imaging position and the motion is not detected between the reference low resolution image and the shift cancel image, In addition, since it is configured to perform reconstruction processing in a high-resolution space using a low-resolution image or shift-cancellation image with a known imaging position, calculation cost is reduced by suppressing unnecessary registration processing in super-resolution processing. It becomes possible to do.

  The second invention may further include a range designating unit that designates a range that can be the processing target area in the low-resolution image.

  According to this, it is possible to effectively suppress useless alignment processing in super-resolution processing by designating in advance a range that requires super-resolution processing.

  The third aspect of the invention includes a first detection unit that determines the presence or absence of movement of the shift cancel image, and a second detection unit that detects a motion vector of the shift cancel image, The alignment processing unit is configured to perform the alignment based on the motion vector.

  According to this, when no motion is detected by the first detection unit, detection of a motion vector by the second detection unit can be omitted, and the calculation cost can be reduced.

  According to a fourth aspect of the present invention, the shift cancel unit generates the shift cancel image by shifting the low resolution image in units of pixels.

  According to this, the shift cancel image can be generated with high accuracy.

  According to a fifth aspect of the present invention, there is provided an imaging system including the image processing device according to any one of the first to fourth aspects of the present invention, and an imaging device that captures the low-resolution image. An image sensor that photoelectrically converts light from a subject and outputs a pixel signal; an optical system that guides light from the subject to the image sensor; a light image that forms an image on a light receiving surface of the image sensor; and the image sensor And an optical shift mechanism that performs a relative circular motion, and a position information acquisition unit that acquires information of the imaging position of the imaging element.

  According to a sixth aspect of the present invention, a plurality of low-resolution images captured by causing the image sensor and a light image formed on the light receiving surface of the image sensor to move relative to each other are acquired, and these images are captured. An image processing method for generating a high-resolution image from a plurality of low-resolution images at different positions, the position of a reference low-resolution image based on the position of the low-resolution image based on information on the imaging position of the low-resolution image A shift cancel generation step for generating a shift cancel image matched with the image, a motion detection step for detecting a motion in the low resolution image by comparing the shift cancel image with the reference low resolution image, and a result of the motion detection A registration processing step for generating an alignment image by aligning the shift cancel image with the reference low-resolution image. And a reconstruction processing step for generating a high-resolution image by reconstruction processing, wherein the reconstruction processing step detects motion in the motion detection step for a predetermined processing target region of the shift cancel image. In this case, the reconstruction process is executed using the alignment image. On the other hand, if no motion is detected in the motion detection step, the reconstruction process is performed using the low resolution image or the shift cancel image. The configuration processing is executed.

  The seventh invention is an image processing program for executing the image processing method according to the sixth invention by controlling the image processing apparatus.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the term “axial direction” indicates the optical axis direction (corresponding to the vertical direction in FIG. 4), and the term “radial direction” is the direction orthogonal to the optical axis (corresponding to the horizontal direction in FIG. 4). It shall be shown. The radial direction can take any angle within 360 ° around the optical axis.

(First embodiment)
FIG. 1 is an overall configuration diagram of a network camera system according to a first embodiment of the present invention. As shown in FIG. 1, a network camera system as an imaging system to which the present invention is applied includes at least one imaging device (here, a network camera) 1 and an image processing device 2. The imaging device 1 and the image processing device 2 are connected via the Internet 3, and imaging data generated by the imaging device 1 is transmitted to, for example, the image processing device 2 that exists in a remote place, and a video is captured by the image processing device 2. Is displayed. Various command signals for controlling the imaging device 1 are transmitted from the image processing device 2 to the imaging device 1.

  The imaging data is transmitted from the imaging apparatus to the image processing apparatus using a so-called Internet protocol such as TCP (UDP) / IP. For example, the imaging data is encrypted and encapsulated, for example, in a VPN (Vertual Private Network). ) Or a network camera system called CCTV (Closed Circuit TV) in which the imaging device 1 and the image processing device 2 are connected in a one-to-one relationship by a dedicated line. .

  FIG. 2 is a block diagram illustrating a schematic configuration of the imaging apparatus and the image processing apparatus illustrated in FIG. 1. As shown in FIG. 2, the imaging device 1 includes an imaging unit 11, an image processing unit 12, a data compression / transmission unit 13, and a shift control unit 14. The imaging unit 11 includes an imaging element 31 that photoelectrically converts light from a subject and outputs an analog pixel signal. The image sensor 31 is a two-dimensional CMOS image sensor. The image sensor 31 has, for example, a quad VGA (1280 × 960 pixels) pixel configuration, a sensor size of 1/3 inch, and a pixel pitch L that is an interval between adjacent pixels is 3.75 μm in both the main scanning direction and the sub-scanning direction. It is. Instead of this, a two-dimensional CCD image sensor may be used for the image sensor 31.

  The analog signal output from the image sensor 31 is converted into a digital signal by the A / D converter 32, and this digital signal is input to the image processing unit 12, where color correction, demosaic processing, gradation correction (γ correction). , YC separation processing and the like are performed and converted into image data. This image data is stored in the data compression / transmission unit 13 by, for example, H.264. The image data is transmitted to the image processing apparatus 2 after being subjected to compression processing such as H.264 or MPEG4.

  As will be described in detail later, the imaging unit 11 includes an optical shift mechanism 35 that relatively slightly displaces the optical image formed on the light receiving surface of the imaging element 31 and the imaging element 31. In the optical shift mechanism 35, an optical member (see reference numeral 51 in FIG. 4) that minutely displaces the optical image is rotated by a magnetic force, and the radial and axial positions of the optical member are respectively set. A position control unit 38 for controlling, a radial magnetic sensor 39 and an axial magnetic sensor 40 for detecting the radial and axial positions of the optical member, respectively, are provided.

  The optical shift mechanism 35 is controlled by the shift control unit 14. The rotating portion of the optical shift mechanism 35 is provided with a magnetized portion (not shown) here, and both magnetic sensors 39 and 40 detect the position information of the magnetized portion and output it to the shift control portion 14. The shift control unit 14 controls the magnetic rotation driving unit 36 based on this position information to rotate the optical member, and controls the position control unit 38 to hold the optical member at a predetermined position.

  The image processing apparatus 2 includes a data reception decoding unit 21, a display unit 22, a storage unit 23, a super-resolution processing unit 24, a period setting unit 25, and an input unit 26. The image processing apparatus 2 is configured by installing necessary application software in an information processing apparatus such as a personal computer or a workstation, or may be a dedicated apparatus such as a CCTV recorder.

  In the image processing device 2, the compressed image data transmitted from the imaging device 1 is received and decoded by the data receiving / decoding unit 21, converted into RGB image data, and displayed in real time by a display or the like. 22 is displayed. Further, the RGB image data is sent to the storage unit 23 composed of a hard disk drive device or the like, temporarily accumulated therein, and can be read from the storage unit 23 and reproduced on the display unit 22 as necessary.

  For example, when a high-resolution image is necessary for verification of traffic accidents, etc., the image data is read from the storage unit 23 and super-resolution processing is performed by the super-resolution processing unit 24 so that the high-resolution image is obtained. (Still image) can be generated and the high resolution image can be displayed on the display unit 22.

  As will be described in detail later, the input unit 26 receives an input of an imaging cycle from the user and sends it to the cycle setting unit 25. The cycle setting unit 25 determines a circular motion cycle based on the imaging cycle sent from the input unit 26 and transmits a command signal related to the circular motion cycle to the imaging device 1. The shift control unit 14 of the imaging apparatus 1 drives the optical member at a rotational speed corresponding to the designated circular motion cycle by operating the optical shift mechanism 35 based on the command signal related to the circular motion cycle.

  Further, the cycle setting unit 25 transmits information related to the imaging timing to the drive circuit 33, whereby a timing generator (not shown) of the drive circuit 33 generates a synchronization pulse for control at a predetermined interval. This synchronization pulse is output to the image sensor 31, thereby determining the start and end of the charge accumulation period (electronic shutter period) of the image sensor 31. Further, the synchronization pulse is also output to the shift control unit 14, whereby the shift control unit 14 can grasp the period during which the image sensor 31 is actually exposed.

  FIG. 3 is a schematic diagram illustrating processing states in the imaging device 1 and the image processing device 2. As shown in FIG. 3, the image sensor 31 is driven by a drive circuit 33, and imaging (sampling) is performed at a constant cycle (hereinafter, imaging cycle) according to a timing signal generated by the drive circuit 33. For example, when generating 30 frame images per second at a frame rate of 30 frames / sec, the imaging cycle is set to about 30 ms.

  The super-resolution processing unit 24 of the image processing apparatus 2 performs super-resolution processing for generating a high-resolution image from a plurality of temporally continuous frame images. In this super-resolution processing, first, the frame image stored in the storage unit 23 is displayed as a still image by frame advance. When a reference image (reference low-resolution image) is designated by the user from among them, a frame image serving as the reference image and a plurality of frame images before and after the frame image are read out from the storage unit 23 and a super-resolution processing unit 24 to perform super-resolution processing.

  As super-resolution processing, for example, ML (Maximum-likelihood) method, MAP (Maximum A Posterior) method, POCS (Projection On to Convex Sets) method, etc. are adopted and realized by executing application software on CPU. Is done. In general, since the super-resolution processing has a large amount of calculation, a part of the processing may be performed using a GPU (Graphics Processing Unit) or dedicated hardware.

  Here, the ML method uses a square error between a pixel value of a low-resolution image estimated from a high-resolution image and an actually observed pixel value as an evaluation function, and a high resolution that minimizes this evaluation function. This is a method of using an image as an estimated image. That is, the ML method is a super-resolution processing method based on the principle of maximum likelihood estimation. The MAP method is a method for estimating a high-resolution image that minimizes an evaluation function obtained by adding probability information of a high-resolution image to a square error. That is, the MAP method is a super-resolution processing method that estimates a high-resolution image as an optimization problem that maximizes the posterior probability by using some foresight information for the high-resolution image. The POCS method is a super-resolution processing method for obtaining a high-resolution image by creating simultaneous equations regarding pixel values of a high-resolution image and a low-resolution image and sequentially solving the equations.

  In these super-resolution processes, first, a high-resolution image is assumed, and based on the point spread function (PSF function) obtained from the assumed high-resolution image from the camera model, the pixel value of the pixel of the low-resolution image is determined. And a process of searching for a high-resolution image that reduces the difference between the estimated value and the observed pixel value (observed value). Therefore, these super-resolution processes are called reconfigurable super-resolution processes.

  Here, the process of searching for a high-resolution image is to search for a position corresponding to a pixel obtained as a low-resolution image in the high-resolution image, which is called “registration”. It is processing. In general, in super-resolution processing, even if the change in pixel position between a plurality of low-resolution images is unknown, it is possible to increase the resolution, so that the alignment processing is repeatedly performed over a wide range around the pixel of interest. For this reason, it is known that the calculation cost becomes extremely high. On the other hand, as described in detail later, in the present invention, the position of the pixel shifted by the optical shift mechanism 35 is known, and each frame image, that is, a low-resolution image is captured at the known position. For at least a stationary subject, much of the alignment processing can be omitted by optical shift, and the calculation cost can be greatly reduced.

  Note that super-resolution processing that uses image information that spans a plurality of temporally continuous frames may be particularly referred to as inter-frame reconstruction type super-resolution processing. On the other hand, when the reconstruction type super-resolution processing is performed within one frame, it is called intra-frame reconstruction type super-resolution. In this embodiment, inter-frame reconstruction type super-resolution is adopted.

  Note that here, the still image that has been increased in resolution by super-resolution processing is reproduced in the image processing device 2, but if the processing capability of the image processing device 2 is sufficiently high, it can be obtained by super-resolution processing. It is also possible to reproduce a moving image using a high-resolution image as a frame image.

  4 is a cross-sectional view showing the imaging unit 11 of the imaging device 1 shown in FIG. 2, and FIG. 5 is an exploded perspective view of the optical shift mechanism 35 of the imaging device 1 shown in FIG. FIG. 7 is a perspective view of the optical shift mechanism 35, FIG. 7 is a plan view of the optical shift mechanism 35, and FIG. 8 is a cross-sectional view of the main part of the optical shift mechanism. In FIG. 5, for convenience of explanation, each member (optical member 51 and the like) accommodated inside the optical capsule (capsule member) 50 is shown in a form taken out to the outside. 5 and 6, the rotational drive coil 81 shown in FIG. 7 is omitted.

  As illustrated in FIG. 4, the imaging unit 11 of the imaging device 1 forms an image on the light receiving surface 31 a of the imaging device 31 with light from a sensor module 41 provided with the imaging device 31 and a subject (not shown). A lens unit (optical system) 42 and an optical shift mechanism 35 that displaces a light image formed on the light receiving surface 31 a of the image sensor 31 in a direction perpendicular to the optical axis C are provided. The lens unit 42 is supported on the substrate 46 via the lens holder 45. The sensor module 41 and the optical shift mechanism 35 are also supported by the substrate 46. Note that other electrical components and the like are mounted on the substrate 46 as necessary.

  The optical shift mechanism 35 mainly includes, for example, an optical capsule 50 formed of a transparent resin or a glass material, and an optical member 51 accommodated in the optical capsule 50. Inside the optical capsule 50, a liquid (fluid) 52 having a refractive index higher than that of air is sealed. Further, the refractive index of the liquid 52 is set to be smaller than the refractive index of a parallel plate 53 described later.

  The optical member 51 has a substantially disk shape, and a parallel plate 53 inclined at a predetermined angle with respect to the optical axis C of the lens unit 42 is provided at the center thereof. The parallel plate 53 can be formed from a material such as optical glass or acrylic resin. Further, on the outer peripheral side of the optical member 51, an annular back yoke 54 that supports the outer peripheral portion, an annular radial magnet 55 attached to the outer peripheral portion of the back yoke 54, and an optical member in the back yoke 54 There are provided an upper magnet (first shaft side magnet) 56 and a lower magnet (second shaft side magnet) 57 respectively fixed to the light incident side and light emitting side portions 51 (here, the upper and lower surfaces of the back yoke 54). It has been. Both the upper magnet 56 and the lower magnet 57 have an annular shape with the same diameter. These members 54 to 57 are arranged coaxially with the optical axis C as the center, and are accommodated together with the optical member 51 in the optical capsule 50 filled with the liquid 52.

  The optical capsule 50 is composed of two members 50a and 50b that are overlapped in the vertical direction, whereby a sealed space in which the liquid 52 is enclosed is formed. The optical capsule 50 has a constricted cross section at the center. The optical capsule 50 is connected to the central portion 60 that defines a disk-shaped space with the optical axis C as the center, and to the outer peripheral side of the central portion 60 and has a rectangular cross section. And an annular portion 61 that defines an annular space. The central portion 60 accommodates the parallel flat plate 53, while the annular portion 61 widened in the vertical direction accommodates the outer peripheral portion of the optical member 51, the back yoke 54, and the like. Here, the distance between the inner surface of the optical capsule 50 and the surface of the optical member 51 is set to 0.5 mm. In the optical capsule 50, at least the optical path portion around the optical axis C in the central portion 60 (the region through which incident light from the lens unit 42 passes) may be formed of the transparent material as described above.

  Here, the distance D between the lens unit 42 in which the optical capsule 50 (the central portion 60) is arranged and the sensor module 41 is usually very narrow (for example, about 3 mm), and the photographing range becomes a wide angle. It becomes narrower accordingly. However, if only the optical member 51 (parallel plate 53) is arranged in the optical path, the apparatus can be configured without any problem even though the distance D is narrow.

  Further, the upper surface 60a of the central portion 60 of the optical capsule 50 is in contact with the lower surface 42a of the lens unit 42, whereby the optical path between the optical capsule 50 and the lens unit 42 (strictly, incident light passes). Space) is sealed. Further, the lower surface 61 a of the annular portion 61 is in contact with the upper surface 46 a of the substrate 46, so that the optical path between the optical capsule 50 and the sensor module 41 (that is, the optical capsule 50, the sensor module 41, and the substrate 46). The defined space V) is sealed. With such a configuration, it is possible to prevent deterioration in image quality due to intrusion of dust or the like into the optical path from the lens unit 42 to the image sensor 31. As a method for sealing the optical path as described above, the optical capsule 50 may be brought into direct contact with the image sensor as long as the imaging element 31 is regarded as a molded image sensor itself.

  Note that portions other than the portion corresponding to the optical path of the optical capsule 50 may be opaque (for example, black). By doing in this way, what is called stray light into which unnecessary light enters into the image sensor 31 can be blocked.

  In this embodiment, an antifreeze liquid (a mixture of polypropylene glycol and water) is used as the liquid 52. When the liquid 52 contains water, the back yoke 54 formed of a magnetic material (for example, an iron-based material) may be rusted. In order to prevent this, a resin coating is applied to the surface of the back yoke 54. Note that the antifreeze liquid does not need to be aqueous, and for example, transparent silicone oil may be employed. In this case, there is no risk of rusting on the diameter side magnet 55 and the like, and rust prevention treatment such as resin coating is not required. Moreover, when the installation environment of the imaging device 1 is limited to a room, it is not always necessary to use antifreeze. Although the magnetic characteristics are slightly inferior, the formation of rust can be suppressed by forming the back yoke 54 from a nonmagnetic metal material such as SUS316 or a resin.

  As shown in FIG. 5, in the optical shift mechanism 35, the optical member 51 and the radial magnet 55 are not in contact with other than the fluid by applying a magnetic force to the radial magnet 55 around the optical capsule 50. The magnetic member 51 is rotated in a non-contact manner with the inner wall of the optical capsule 50, and the magnetic force is applied to the radial magnet 55, the upper magnet 56, and the lower magnet 57. And three position control units 38 that hold the axial position at a regular position (a fixed position where the parallel plate 53 can rotate without contacting the optical capsule 50) As shown in FIG. 7, it is called position control part 38a, 38b, 38c.). As described above, the diameter side magnet 55, the upper magnet 56, and the lower magnet 57 of the optical member 51 are all permanent magnets, and an electromagnet is employed in the position control unit 38 that applies a magnetic force to them. No signal line is drawn into the capsule 50, and the sealing property of the optical capsule 50 can be improved.

  As will be described in detail later, the rotational drive and radial position control of the optical member 51 are performed by three radial magnetic sensors 39 (hereinafter, when it is necessary to distinguish each of them) as the position detector shown in FIG. As shown in FIG. 7, this is performed based on information output from the radial magnetic sensors 39a, 39b, and 39c. Further, the axial position control of the optical member 51 is performed based on information output by the axial magnetic sensor 40 as the position detection unit shown in FIG.

  With such a configuration, the optical member 51 rotates in a non-contact state within the optical capsule 50 filled with the liquid 52. The configuration using this magnetic force is similar to the configuration of a so-called bearingless motor, there is no sliding portion, the optical shift mechanism 35 is driven with extremely low vibration, and a long life can be achieved. it can.

  As shown in FIG. 6, an origin sensor 65 that periodically detects the origin position accompanying the rotation of the optical member 51 is provided around the optical capsule 50. The origin sensor 65 is composed of a reflection type photosensor (photoreflector), and detects a marker 66 (see FIG. 4) provided on the upper side of the diameter side magnet 55. The origin sensor 65 is not limited to a reflective photosensor, and a known sensor including another optical sensor can be used. The parallel flat plate 53 inclined with respect to the optical axis C is detected by the magnetic pole position of the magnetized portion 68 (see FIG. 7) of the radial magnet 55 detected by the output of the origin sensor 65 and the output of the radial magnetic sensor 39. It is possible to reliably grasp the light shift direction due to.

  As shown in FIG. 4, the marker 66 is disposed so as to face the origin sensor 65 with the optical capsule 50 interposed therebetween. The marker 66 is disposed on an extension line in the inclination direction of the parallel plate 53, and as indicated by an arrow A, the light incident on the parallel plate 53 is shifted in the direction in which the marker 66 is attached. Therefore, the origin sensor 65 can detect the shift direction of incident light by the parallel plate 53 (that is, the rotational position of the optical member 51). The marker 66 is optically identified. For example, the marker may be printed on the radial magnet 55 in a color different from that of the magnet, or the surface roughness of the magnet may be changed to obtain a different reflectance as a result. It may be made to be. The origin sensor 65 detects the marker 66 using the difference in reflectance between the marker 66 and the diameter side magnet 55.

  In the rotation direction of the optical member 51, the relationship between the inclination direction of the parallel plate 53 and the position of the marker 66 in the rotation direction is grasped in advance, so that the direction of the parallel plate 53 (that is, optical) at the timing when the marker 66 is detected. The direction of the target shift). That is, since the rotation angle of the parallel plate 53 can be known when the marker 66 is detected, the direction in which the optical shift is performed can be specified. Here, the optical shift amount by the parallel plate 53 depends on the angle of the parallel plate 53 with respect to the optical axis, but this angle is detected by the axial magnetic sensor 40 and is made constant by the magnetic circuit shown in FIG. Be controlled. Further, since the rotating body including the back yoke 54 is rotated, the shaft shake is suppressed to be small due to the gyro effect similarly to the rotation of the frame, so that the angle of the parallel plate 53 with respect to the optical axis (that is, the optical shift amount) is approximately. It becomes constant. Therefore, the coordinates of the optical image shift due to the optical shift can be calculated from the rotation angle of the parallel plate 53.

  By the way, in the super-resolution processing performed by the super-resolution processing unit 24 shown in FIG. 2, the imaging position is on the order of submicron (for example, the pixel pitch L is 5.6 μm, and this is set to 4 by super-resolution processing. In the case of enlargement to x4, the newly generated pixel pitch L is 5.6 / 4 = 1.4 μm, and displacement of sub-micron order is necessary). It is known that the calculation cost can be significantly reduced. In the imaging apparatus and network camera system according to the present invention, the origin sensor 65 as described above is employed, and the light when the imaging device 31 performs imaging in accordance with the timing signal (synchronization pulse) generated by the drive circuit 33. Since the image shift position is determined, the calculation cost of the super-resolution processing can be reduced. That is, according to the present invention, since the shift position of the image can be reliably grasped, the search amount of the above-described high-resolution image is reduced, and the calculation cost of the super-resolution processing for the still portion of the imaging target of the optical shift is reduced. Is greatly reduced.

  As shown in FIG. 7, the radial magnet 55 includes a magnetized portion 68 including 16 magnetic poles alternately magnetized to the N and S poles along the circumferential direction. The diameter-side magnet 55 is a so-called plastic magnet formed of polyphenylene sulfide resin (PPS) in which minute magnetic particles are dispersed and mixed, thereby preventing water absorption and swelling even in the liquid 52 containing water. Is done. Here, neodymium is adopted as the magnetic particles. A neodymium magnet having an extremely large magnetic force can obtain a large driving torque, and thus is effective when the viscosity of the liquid 52 becomes large at low temperatures. On the other hand, since neodymium can be oxidized by water and cause rust, the surface of the diameter-side magnet 55 is coated with a resin material to prevent contact with the liquid 52. Similarly, the upper magnet 56 and the lower magnet 57 are made of PPS in which neodymium magnetic particles are dispersed and mixed, and these are also coated with a resin. The magnetic particles used as the diameter-side magnet 55, the upper magnet 56, and the lower magnet 57 are not limited to neodymium, and for example, ferrite, samarium cobalt, or the like can also be used.

  The magnetic rotation drive unit 36 includes a stator core 80 formed by stacking a plurality of electromagnetic steel strips, and a rotation drive coil 81 formed of a conductive wire wound around the stator core 80. The substantially U-shaped stator core 80 is disposed at equal intervals along the outer peripheral surface in a state where both ends are in contact with the outer wall of the optical capsule 50, and the diameter-side magnet 55 is interposed via the optical capsule 50. Opposed to the magnetized portion 68. Thereby, the optical shift mechanism 35 has the same configuration as a so-called inner rotor type three-phase motor (12 slots, 16 poles in this embodiment). The magnetic rotation driving unit 36 rotates the magnetizing unit 68 in a predetermined direction by an attractive force and a repulsive force acting on the magnetizing unit 68 by a magnetic field generated by passing a current through the rotation driving coil 81. As a result, the back yoke 54 and the optical member 51 (parallel plate 53) rotate around the optical axis C.

  The radial magnetic sensors 39a, 39b, and 39c are composed of Hall elements, and are arranged at equal intervals in the circumferential direction along the outer peripheral surface in a state where the radial magnetic sensors 39a, 39b, and 39c are in contact with the outer wall of the optical capsule 50. It is located between the two magnetic rotation driving parts 36. With such arrangement of the radial magnetic sensors 39a, 39b, 39c, the radial displacement amount of the optical member 51 can be accurately detected. By using a Hall element as the radial magnetic sensor 39, the position of the optical member 51 in the optical capsule 50 can be detected with an accuracy of about 1 μm. If higher accuracy is required, an optical position sensor (PSD: Position Sensitive Detector) may be used instead of the Hall element. By using PSD, the position detection accuracy is improved to about 0.1 μm.

  Note that there is no contradiction between the point that the detection accuracy when the Hall element is used as the position detection means is 1 μm and the above-described “determine the imaging position in submicron order”. In the optical shift mechanism 35 of this embodiment, the optical member 51 (parallel plate 53) has only power for shifting incident light in parallel, and the shift position of the optical image is determined solely by the rotation angle of the optical member 51. The Even if the optical member 51 is displaced parallel to the optical axis direction or the radial direction, the shift position by the parallel plate 53 is not affected at all, and if it is affected, the inclination angle of the optical member 51 with respect to the optical axis C is not affected. This is a case where fluctuates. In this embodiment, as will be described in detail later, the influence of the tilt angle variation is reduced, and as a result, even if the detection accuracy is about 1 μm, the shift position is determined with submicron order accuracy.

  In the configuration of the 12-slot 16-pole shown in FIG. 7, six spaces are generated between the adjacent magnetic rotation driving units 36 on the outer periphery of the optical capsule 50. Therefore, the three radial magnetic sensors 39a, 39b, 39c can be arranged in these spaces at positions rotated by 180 degrees around the three position control units 38a, 38b, 38c and the optical axis C, respectively. Accordingly, it is possible to easily control the position of the optical member 51 in the radial direction while effectively reducing the size of the apparatus by effectively using the space outside the optical capsule 50.

  The radial magnetic sensor 39 detects the magnetic force generated from the magnetized portion 68 of the radial magnet 55, thereby outputting position information of the magnetic poles (N pole and S pole) of the magnetized portion 68. More specifically, when one set of the N pole and the S pole of the radial magnet 55 moves relative to the radial magnetic sensor 39 relative to itself, the shift control unit 14 uses a sine wave having this as one cycle as position information. (See FIG. 2). The shift control unit 14 calculates the rotation speed of the optical member 51 by processing this phase information. That is, the radial magnetic sensor 39 and the shift control unit 14 function as a rotation position detection unit and a rotation speed detection unit of the optical member 51.

  Instead of such a method of detecting the rotational speed, for example, marking is performed on the outer peripheral portion (magnetized portion or the like) of the optical member 51 or the diameter-side magnet 55, and this is detected by an optical sensor (photo reflector). May be. In this case, if the black-and-white marking is used, the rotation angular velocity can be detected at a higher sampling rate because drawing can be performed at a relatively narrow pitch.

  The axial magnetic sensor 40 is disposed in the vicinity of the position control unit 38 in contact with the upper outer wall of the optical capsule 50. The axial magnetic sensor 40 is disposed so as to face the upper magnet 56 via the optical capsule 50 as shown by a two-dot chain line in FIG. The axial magnetic sensor 40 may be disposed on the front side. The axial magnetic sensor 40 detects the magnitude of the magnetic force generated from the upper magnet 56, and outputs axial position information of the back yoke 54 (ie, the optical member 51) to which the upper magnet 56 is attached. Based on this position information, the shift control unit 14 (see FIG. 2) controls the current flowing through the axial position control coil 71 (see FIG. 8), and controls the position of the optical member 51 in the axial direction.

  As shown in FIG. 8, the position control unit 38 protrudes upward (in a direction parallel to the optical axis direction) at the lower end portion of the stator core 70 having a substantially E shape in which a plurality of electromagnetic steel strips are stacked. An axial position control coil 71 wound around the protrusion 70a and a radial position control coil 72 wound around a protrusion 70b protruding laterally (in a direction perpendicular to the optical axis direction) at the intermediate portion of the stator core 70. And have. The tips of the protrusions 70a and 70b are in contact with the lower surface and the outer peripheral surface of the optical capsule 50, respectively.

  The electromagnet 74 formed on the protrusion 70 a by the axial position control coil 71 functions as an axial magnetic force generation unit that generates a magnetic force acting on the lower magnet 57 and is used for axial position control of the optical member 51. It is done. In addition, the electromagnet 75 formed on the protrusion 70 b by the radial position control coil 72 functions as a radial magnetic force generation unit that generates a magnetic force acting on the radial magnet 55, and the radial position of the optical member 51. Used for control.

  In addition, a protrusion 70 c that protrudes downward (in a direction parallel to the optical axis direction) is provided at the upper end of the stator core 70 so as to face the protrusion 70 a with the optical capsule 50 interposed therebetween. The protrusion 70 c is formed as a part of the permanent magnet 76, and its tip is in contact with the upper surface of the optical capsule 50. The permanent magnet 76 functions as an axial magnetic force generating means for generating a magnetic force acting on the upper magnet 56, cooperates with the lower electromagnet 74 facing thereto, and is used for axial position control of the optical member 51. In this case, the permanent magnet 76 and the upper magnet 56 have the same magnetic pole (here, both N poles), and a repulsive force acts between them. In addition, although the neodymium magnet is employ | adopted as the permanent magnet 76, considering the balance with the magnetic force by the lower electromagnet 74, you may use another magnet (for example, a ferrite magnet etc.).

  The protrusion 70 b is formed as a part of a connecting member 77 that connects the upper and lower portions of the stator core 70. The connecting member 77 is made of a non-magnetic material (for example, resin, ceramic, etc.) in order to prevent magnetic interaction between the electromagnet 74, the electromagnet 75, and the permanent magnet 76.

  In the present embodiment, the position control unit 38 functions as means for controlling the radial position and the axial position of the optical member 51. However, the position control unit 38 controls the radial position and the axial position. It is also possible to employ a configuration in which the position control units are individually provided.

  In the axial position control, the electromagnet 74 is excited by passing a current through the axial position control coil 71, and a repulsive force is generated between the electromagnet 74 having the same magnetic pole (here, the S pole) and the lower magnet 57. . The position of the optical member 51 in the axial direction is balanced by balancing the magnitude of this repulsive force with the repulsive force acting between the permanent magnet 76 located on the opposite side of the electromagnet 74 and the upper magnet 56 across the optical capsule 50. Is held in the normal position.

  On the other hand, when the electromagnet 74 is not excited, the repulsive force from the permanent magnet 76 acts on the optical member 51, the optical member 51 moves downward, and the attractive force between the lower magnet 57 and the non-energized stator core ( That is, the lower magnet 57 is fixed in a state where the lower end of the lower magnet 57 is in contact with the optical capsule 50. As described above, when the image pickup apparatus is not turned on (for example, when the image pickup apparatus is transported or the like), the optical member 51 is securely fixed, so that the parallel plate 53 can be prevented from being damaged. . In this case, it is preferable that the optical member 51 (parallel flat plate 53) does not contact the optical capsule 50 from the viewpoint of preventing damage even in the fixed state. Therefore, when the optical member 51 is in the normal position shown in FIG. 8, the distance d1 between the lower magnet 57 and the optical capsule 50 is set to be smaller than the distance d2 between the optical member 51 and the optical capsule 50. The

  Note that the optical shift mechanism 35 may be configured such that an attractive force acts between the permanent magnet 76 and the upper magnet 56 and between the electromagnet 74 and the lower magnet 57 to balance them. However, by applying a repulsive force as described above, a frictional force is generated in the optical member 51 due to the influence of the magnetic force (attraction force) by the upper magnet 56 or the lower magnet 57 at the time of initial startup or the like, and the apparatus cannot be started. There is an advantage that troubles can be prevented.

  In the radial position control, the electromagnet 75 is excited by passing a current through the radial position control coil 72, and a repulsive force is generated between the radial magnet 55. This repulsive force acts from three directions around the radial side magnet 55 due to the arrangement of the position control units 38a, 38b, and 38c shown in FIG. 7, and by balancing them, the radial position of the optical member 51 is fixed. Retained.

  In the optical shift mechanism 35, as shown in FIG. 7, the position control unit 38a and the radial magnetic sensor 39a, the position control unit 38b and the radial magnetic sensor 39b, and the position control unit 38c and the radial magnetic sensor 39c are Are arranged at positions rotated by 180 ° around the optical axis C. The radial position control coils 72 (see FIG. 8) of the position controllers 38a, 38b, and 38c are driven based on the position information detected by the radial magnetic sensors 39a, 39b, and 39c, respectively. That is, the radial position control coils 72 of the placement control units 38a, 38b, and 38c are driven based on position information detected at positions facing each other across the optical capsule 50, respectively. With such a configuration, the increase in the output of the radial magnetic sensor 39 is directly related to the increase in the separation distance between the corresponding position control unit 38 and the optical member 51 (diameter side magnet 55), and the position control process is performed. Becomes easy.

  Further, in the optical shift mechanism 35, as shown in FIGS. 6 and 7, the magnetic rotation driving unit 36, the radial magnetic sensor 39, the axial magnetic sensor 40, and the position control unit 38 are each provided in the optical capsule 50. It is made to contact with the outer surface. In this way, the components necessary for the rotational drive control and position control of the optical member 51 are arranged in contact with the optical capsule 50. Thereby, if the dimensional accuracy of the optical capsule 50 itself is managed well, the positional relationship (that is, the distance relationship) of each component can be determined with extremely high accuracy, and high control performance can be realized. Here, as a preferred example, all the components necessary for the rotation drive control and the position control are in contact with the optical capsule 50. However, the present invention is not limited to this, and some of these components contact the optical capsule 50. A configuration that touches is also possible.

  Further, in the optical shift mechanism 35, an example in which a configuration corresponding to a so-called three-phase motor is applied to rotate the optical member 51 is shown, but the same applies to a configuration similar to a single-phase motor. An effect can be obtained.

  FIG. 9 is a cross-sectional view showing a state of incidence of light on the image sensor. FIG. 9A shows a state where the optical path of the incident light is shifted to the rightmost side, and FIG. ) Shows a state in which the parallel flat plate 53 is rotated 180 °. When the parallel flat plate is further rotated 180 ° from the state shown in FIG. 9B, the state returns to the state shown in FIG.

  As shown in FIG. 9, since the parallel flat plate 53 of the optical member 51 is inclined with respect to the optical axis C of the lens unit 42, the light incident through the lens unit 42 is refracted and the light receiving surface of the image sensor 31. When the position of the light incident on the optical plate 51 changes according to the rotational position of the parallel plate 53 and the optical member 51 is rotated by the optical shift mechanism 35, a light image formed on the light receiving surface of the image sensor 31 becomes an optical member. It moves so as to draw a circle with a period (circular motion period) corresponding to the rotational speed of 51, and thereby the optical image can be relatively displaced relative to the image sensor 31.

  FIG. 10 is a configuration diagram of the shift control unit 14 in the optical shift mechanism 35 shown in FIG. Below, the rotational drive control and position control of the optical member 51 by the shift control part 14 are demonstrated using FIG. 10 and FIG. 7 together.

  First, the rotational drive control of the optical member 51 will be described. As shown in FIG. 7, the diameter side magnet 55 of the optical member 51 is provided with 16 poles (8 pairs of N poles and S poles), and the magnetism generated by the magnetized parts 68 is provided. Are detected by three radial magnetic sensors 39 arranged at intervals of 120 ° in the circumferential direction. The detection output (analog signal) of the radial magnetic sensor 39 is input to the comparator 91a. Before the input, the signal level is changed to ± with respect to the ground level by an amplifier and an analog offset circuit (not shown). It is shifted like analog. The comparator 91a outputs an FG signal (pulse signal) when detecting a zero cross of the analog signal. This FG signal is input to the arithmetic processing unit 92 as an interrupt signal (IRQ). As described above, since there are three radial magnetic sensors 39, the arithmetic processing unit 92 receives an interrupt signal based on the zero cross for each sensor. The detection output of the origin sensor 65 is converted into a binary pulse signal by being compared with a predetermined level in the comparator (CMP) 91b, and then output to the arithmetic processing unit 92 as an interrupt signal (IRQ). Is done. Further, the synchronization pulse output from the drive circuit 33 is input as an interrupt signal (IRQ) to the arithmetic processing unit 92 of the shift control unit 14.

  Here, since the interval between the magnetic poles of the magnetized portion 68 is set to be the same, the distance between the magnetic poles is known. Therefore, the speed calculation value Vn can be obtained by dividing this distance by the time between pulses counted by the high-speed counter provided in the shift control unit 14 (for example, the interval between adjacent zero crosses). That is, in this embodiment, the arithmetic processing unit 92 also functions as a speed calculation unit, and calculates the rotation speed of the optical member 51 based on the output of the radial magnetic sensor 39. With such a configuration, it is possible to detect the speed and position of the optical member 51 with the radial magnetic sensor 39, and the cost of the apparatus can be reduced.

Here, details of the processing in the arithmetic processing unit 92 will be described. The arithmetic processing unit 92 includes a CPU 93 that executes arithmetic processing, a ROM 94 that stores a program necessary for the processing of the CPU 93, and a RAM 95 that functions as a work area when the CPU 93 executes the program. The arithmetic processing unit 92 rotates the optical member 51 at a constant angular speed by performing PI control (proportional integral control) based on the calculated rotation speed of the optical member 51. Since the interval (time) of the FG pulse when the optical member 51 rotates at a constant speed is known, this is the speed theoretical value Vr (set angular speed, that is, speed target value), and the speed relative to the speed theoretical value Vr. The error δV of the calculated value Vn is calculated as follows:
δV = Vr−Vn

The speed theoretical value Vr and the speed calculated value Vn are originally real numbers, but can be normalized in advance as, for example, a 16-bit integer value in order to speed up the calculation by the CPU 93. The P term, which is a proportional term in the PI control, is calculated as follows by multiplying δV by an appropriate gain Gp.
P = Gp × δV

Here, if the gain is infinite, speed control can be performed using only the P term. Generally, the gain is finite, and at this time, a speed offset is generated. Therefore, the error δV is integrated and appropriate. The integral gain I is multiplied to calculate an integral term I (Integral) term as in the following equation.
I = Gi × Σ (δV)

  This P + I is a so-called speed command value, and the CPU 93 sends the speed command value to the pulse width modulator (PWM) 96. Based on this speed command value, the pulse width modulator 96 calculates an ON duty ratio in a predetermined period, and a signal (PWM signal) modulated based on the duty ratio is output to the three-phase driver 97. The three-phase driver 97 includes three systems of push-pull type transistor circuits 98 inside, and controls the current flowing through each rotation drive coil 81 that is star-connected based on the PWM signal output from the shift control unit 14 to rotate. Drive. Here, the common signal extracted from each rotary drive coil 81 is input to the three-phase driver 97, and the phase excitation (ON / OFF) of each push-pull transistor circuit 98 is controlled. The rotation of the optical shift mechanism is controlled by switching the phase excitation. For this current control, a known voltage regulator may be used instead of the pulse width modulator 96. Whichever configuration is employed, the optical member 51 can be rotated at a plurality of different constant angular velocities by changing the speed target value used as the speed theoretical value Vr. As a result, the rotational drive control is executed in a circular motion cycle instructed from the cycle setting unit 25 (see FIG. 2).

  Next, position control of the optical member 51 will be described. As described above, the position detection of the optical member 51 is performed by the three axial magnetic sensors 40 and the three radial magnetic sensors 39 that also serve as input sources of FG signals (speed information). These sensor groups detect the magnetism of the magnetized portions provided in the upper magnet 56 and the diameter-side magnet 55, respectively. This detected value is converted into a digital signal by the A / D converters 101a and 101b. The outputs of the A / D converters 101a and 101b are 8 bits, and the CPU 93 of the arithmetic processing unit 92 receives a total of 8 × 3 × 2 bits of position information signals.

  Then, the CPU 93 calculates a deviation from a theoretical value (positional relationship (known value) between the radial magnetic sensor 39 and the axial magnetic sensor 40 when the optical member 51 is in the normal position) based on the acquired position information signal. A correction value to be corrected is calculated, and this correction value (digital signal) is sent to the D / A converters 102a and 102b. In the D / A converters 102a and 102b, the digital signal is converted into an analog signal. Based on this analog signal, the radial drive driver 99 and the axial drive driver 100 drive the radial position control coil 72 and the axial position control coil 71, respectively, to position the optical member 51 in the optical capsule 50. Control.

  In the present embodiment, the outputs of the radial magnetic sensor 39 and the axial magnetic sensor 40 directly indicate distance information. Therefore, calculation only needs to be performed for the I term in the PI control.

  Next, details of a method for detecting the rotational position of the optical member in the above-described rotational drive control and position control will be described. FIG. 11 is a diagram showing detection results of the marker 66 and the magnetized portion 68 by the origin sensor 65 and the radial magnetic sensor 39, and FIG. 12 shows a position detection method by the shift control unit 14 based on the detection result of FIG. It is explanatory drawing which shows a detail. FIG. 12 shows changes in the origin sensor output, the high-speed counter value, and the zero cross count value with respect to the elapsed time t on the horizontal axis.

  In the magnetized portion 68 (see FIG. 7), the magnetized pattern is set so that the magnetic force of 16 poles is detected as a sine wave as shown in FIG. 11 by the radial magnetic sensors 39a to 39c. Then, the comparator 91a (see FIG. 10) performs zero crossing (in FIG. 11) when the magnetic pole changes from the N pole to the S pole (or from the S pole to the N pole) in the detection outputs of the respective radial magnetic sensors 39a to 39c. When an FG signal is detected, an FG signal (pulse signal) is output, and this FG signal is input to the arithmetic processing unit 92 as an interrupt signal (hereinafter referred to as “zero cross interrupt signal”).

  Zero crossing is detected 16 times per rotation of the optical member 51 (see FIG. 7). For example, when the rotation cycle of the optical member 51 rotating at a constant speed is 225 ms, one radial magnetic sensor 39 detects a zero cross at a cycle of 225/16 = 14.0625 ms. Further, since the interval between the magnetic poles of the magnetized portion 68 is set to be the same (that is, the interval of 22.5 ° in the circumferential direction), the detection output of each radial magnetic sensor 39 is as shown in FIG. The phase is shifted by 1/3 period. Therefore, the three radial magnetic sensors 39a to 39c detect 48 (16 × 3) zero crosses every time the optical member 51 rotates (225 ms) as a whole, and the theoretical value of the detection cycle is 4.6875 ms. (225 ms / 48). Here, the displacement angle between the zero crosses is 7.5 ° (360 ° / 48).

  Further, as shown in FIG. 11, the origin sensor 65 (see FIG. 10) outputs a pulse signal at a cycle of 225 ms, and the rising edge of this pulse signal is an interrupt signal for the CPU 93 (see FIG. 10) of the arithmetic processing unit 92. Functions as (IRQ).

  The CPU 93 can recognize the rotation of the optical member 51 by 7.5 ° by detecting the zero cross, but in this embodiment, as shown in FIG. 12, the elapsed time between the zero crosses is 16 bits built in the CPU 93. Thus, the rotational position of the optical member 51 is detected with higher accuracy.

  In FIG. 12, when the CPU 93 detects a zero cross interrupt signal (IRQ2) immediately after the interrupt signal (IRQ1) based on the output of the origin sensor 65, the CPU 93 resets the zero cross count value, which is a program variable, to zero. Thereafter, the CPU 93 counts up the zero cross count value every time it detects a zero cross interrupt signal (IRQ3, IRQ4, etc.). The zero cross count value is reset to 0 at the time of detection. That is, the zero cross count value is periodically reset and counted up between 0 and 47. The CPU 93 can grasp the rotation angle (rotation position) of the optical member 51 with higher accuracy by monitoring the value of the zero cross count value.

Zero-cross interrupt signals (IRQ1 to IRQ4, etc. in FIG. 12) directly function as reset pulses for the high-speed counter. That is, the high-speed counter is reset to 0 by each zero cross interrupt signal, and repeats counting up again. Here, if the count clock of the high-speed counter is 1 MHz, for example, and the period of the zero-cross interrupt signal is 4.6875 ms, 4687 (4.6875 × 10 ⁻³ × 10 ⁶ ) counts between adjacent zero-cross interrupt signals. Is called. That is, the count value of the high-speed counter takes 0 to 4687 as a theoretical value.

  In addition, the shift control unit 14 (see FIG. 10) receives an interrupt signal (IRQ5) instructing the start (or end) of the charge accumulation period from the drive circuit 33 in FIG. In FIG. 12, when the IRQ5 occurs, the zero cross count value is 1, and when the value of the high-speed counter acquired by the CPU 93 is 2100 at this time, the rotation angle θrot of the optical member 51 is 10.86 ° ( (1 + 2100/4687) × 7.5 °). Here, the rotation angle of the optical member 51 at the start of the charge accumulation period is detected, but the optical member 51 continues to rotate while the charge accumulation is performed. Since this rotation speed is constant and known, for example, the rotation position at the intermediate point between the start and end of the charge accumulation period may be obtained according to the charge accumulation period and used as a representative value.

  As described above, the direction in which the optical shift is performed is based on the detection position of the marker 66 by the origin sensor 65. However, as shown in FIG. Has an offset (time difference). By intentionally providing such an offset, contention between the interrupt (IRQ1) based on the origin sensor 65 and the interrupt due to zero crossing (IRQ2) is avoided.

  On the other hand, the accurate rotational position of the optical member 51 cannot be calculated unless this offset is taken into consideration. That is, the rotation angle θrot represents the angle at which the optical member 51 has rotated from the first zero cross (IRQ2) generated after the origin sensor 65 detects the marker 66 to the start of the charge accumulation period. The optical shift is performed by adding an angle due to an offset to θrot. However, when the zero cross count value is 47, the rotation angle may exceed 360 ° in consideration of the offset. In this case, 360 ° may be subtracted from the calculation result.

  In addition, the angle by the offset is measured in advance by measuring the time difference (for example, 2 ms) from the detection of the marker 66 by the origin sensor 65 to the detection of the first zero cross (the zero cross obtained by processing the output of the radial magnetic sensor 39c). Thus, it can be calculated as 3.2 ° (2/225 × 360 °) from the theoretical rotation period of the optical member (here, 225 ms).

  In some cases, the output of the origin sensor 65 can be used as a reset pulse of the high-speed counter. In this case, the offset value is set by using the next zero-cross interrupt signal as event information in the input capture function of the CPU 93. It is also possible to update. This offset update process may be performed, for example, when the imaging apparatus is turned on or when a temperature change within a predetermined period is large.

  The optical shift by the optical shift mechanism 35 (see FIG. 2) is performed such that the optical image draws an arc on the imaging surface of the imaging element 31, and this arc radius (hereinafter referred to as “shift radius”). ) Is uniquely determined by the inclination angle of the parallel plate 53. Accordingly, since the rotation angle θrot can be measured with high accuracy as described above, position information (X, Y of the optical image shifted by the optical shift) is obtained by performing a simple trigonometric function calculation. (Coordinate position) can be calculated with high accuracy. More specifically, the position information is calculated with a so-called sub-pixel accuracy that is finer than a pixel (pixel) in a high-resolution space to be obtained by super-resolution processing. As described above, the acquired optical shift position information is sent from the shift control unit 14 to the storage unit 23 of the image processing apparatus 2 and stored in association with the captured frame image. In FIG. 2, the shift position information is transmitted to the storage unit 23 via a route separated from the image data. For example, the shift position information is transmitted from the shift control unit 14 to the image processing unit 12, and the image processing unit 12, the image data may be packed and transmitted.

  Thus, the relative rotation position (rotation angle) of the optical member 51 can be detected by detecting the zero cross. In the configuration in which the rotating body is magnetically held in the liquid as in the present embodiment, the distance between the magnetic pole on the rotating body and the magnetic sensor is changed according to the change in the distance between the radial magnetic sensor 39 and the radial magnet 55. The probability of changing is high. However, since the zero cross is used for detecting the rotational position of the optical member 51, it is less susceptible to the influence of the distance between the radial magnetic sensor 39 and the radial magnet 55 (that is, less susceptible to the influence of distance fluctuation at the portion where the magnetic pole is switched). There is.

  FIG. 13 is a schematic diagram illustrating a situation of relative circular motion of pixels with respect to an optical image. Here, an example is shown in which the shift radius r is set to the same size as the pixel pitch of the image sensor 31 in the optical shift mechanism of the image pickup apparatus. The relative circular motion of the pixel with respect to the optical image is such that the optical image is displaced with respect to the pixel of the fixed image sensor 31 as shown in FIG. Is illustrated so that the pixels of the image sensor 31 move with respect to a stationary light image. Furthermore, each pixel has a light receiving region of a predetermined size for receiving light. However, in the following description, for convenience, only the center position (pixel center) of each pixel is a plurality of straight lines indicated by solid lines in FIG. It is illustrated as an intersection of.

  In FIG. 13, the X axis indicates the main scanning direction, and the Y axis indicates the sub scanning direction. In addition, a plurality of straight lines in the X-axis direction and the Y-axis direction indicated by solid lines in the drawing are arranged at intervals of the pixel pitch. Further, a plurality of straight lines in the X-axis and Y-axis directions indicated by broken lines in the drawing are for indicating pixels in the high resolution space in the super-resolution processing described above. Here, pixels in the high resolution space Is defined as a sub-pixel obtained by dividing the pixel pitch into four for both main scanning and sub-scanning (that is, when super-resolution processing of 4 × 4 times is performed). The advantageous effect obtained by the 4 × 4 times super-resolution processing is when the number of divisions is smaller than this (for example, the pixel pitch L is divided into three, ie, 3 × 3 times super-resolution). Is obtained in the same manner.

  The imaging device 31 has a so-called Bayer array in which R pixels that receive an R (Red) component of incident light, B pixels that receive a B (Blue) component, and G pixels that receive a G (Green) component. 1 is a so-called single-plate color image sensor. In this Bayer array, G pixels are arranged in a zigzag pattern (checker flag shape) with the number of pixels being 1/2 of the total number of pixels, and R pixels and B pixels are G with a number of pixels that is 1/4 of the total number of pixels. The pixels are dispersedly arranged at positions other than the pixel arrangement position.

  The shift radius r can be changed by adjusting the thickness and inclination angle of the optical path portion in the parallel plate 53 and the polypyrene glycol concentration. For example, when the shift radius r is the same as the pixel pitch, the parallel plate 53 has a thickness of 0.1 mm, an inclination angle of inclination angle = 20 °, and a polypyrene glycol concentration of 60 wt% (60 wt% (the mixing ratio of PG and water). Is 6: 4, and the refractive index is about 1.38).

  As indicated by a solid circle in FIG. 13, the pixel center of each pixel (for example, G pixel) moves in a circular orbit having a radius the same as the pixel pitch with each black circle (●) as the rotation center. To do. Here, imaging is performed while continuously performing relative circular motion of the pixels with respect to the optical image at a constant speed in one direction. Imaging reference positions P1 to P15 indicated by circles in FIG. 13 indicate the timing of imaging, and one frame image is sequentially generated at each of the imaging reference positions that are slightly shifted. In particular, here, the center position of a pixel (for example, G pixel) that moves along a circular orbit is shown as an imaging reference position (substantial imaging position), and charge accumulation starts at each imaging reference position, and imaging immediately after Charge accumulation is completed before the reference position and a pixel signal is output.

  The rotational speed of this circular motion is stably maintained by the above-described PI control, the reference of the rotational position of the parallel plate 53 (see FIG. 9 etc.) is managed by the above-described origin sensor 65 (see FIG. 6), and the parallel plate Since the influence of the inclination angle variation of 53 on the shift width (shift position) is suppressed to be small, the imaging position of the frame image captured at each timing (that is, the relative position between the imaging element and the optical image) is extremely high accuracy. Will be grasped.

  Hereinafter, description will be made with reference to FIGS.

  Information regarding the above-described imaging position (hereinafter simply referred to as “position information”) is sequentially generated by the arithmetic processing unit 92 based on the outputs of the radial magnetic sensor 39 and the origin sensor 65 (see FIG. 10), and the arithmetic processing unit 92 is output from the shift control unit (position information acquisition unit) 14 including 92 to the storage unit 23, and is stored in association with image data in units of frames output from the imaging unit 11. In the course of the super-resolution processing, the position information is referred to by the super-resolution processing unit 24, and the alignment processing is simplified at this time (both refer to FIG. 2).

  In order to achieve appropriate high resolution by super-resolution processing, it is desirable that the imaging timings of all the pixels be uniform, and in this embodiment, a global time in which there is no time difference in charge accumulation timing between pixel lines. A shutter system was adopted.

  In addition, by performing many imaging (sampling) while the pixel performs one circular motion, the quality of the high resolution image obtained by the super-resolution processing can be improved. Set to a non-integer multiple of the imaging period. In this way, by repeating the circular motion, it is possible to capture images at a number of different positions, so a large number of images with slightly different imaging positions can be generated, so the high resolution obtained by super-resolution processing The image quality can be improved. On the other hand, if the circular motion cycle is an integral multiple of the imaging cycle, the imaging reference position does not change even if the circular motion is repeated, and the number of imaging reference positions that can be set by one circular motion is limited.

  Hereinafter, an example of the imaging reference position will be described with reference to FIG. 14 by specifically determining the ratio between the circular motion period and the imaging period. FIG. 14 is a schematic diagram illustrating the state of the imaging reference position in an example of the ratio between the imaging cycle and the circular motion cycle. In FIG. 14, the pixel pitch is shown as 1.

  In the example shown in FIG. 14, the circular motion period is set to 7.5 times the imaging period. Here, if the imaging cycle is 30 ms (about 30 frames / s), for example, the circular motion cycle is 225 ms (= 30 ms × 7.5). In this case, the imaging reference position returns to the original position at the second round motion, and 15 times of imaging (sampling) is performed while the circular motion is performed twice. The respective imaging reference positions are separated by a relative angle of 48 ° (= 360 ° / 7.5). Note that since the super-resolution effect is suppressed when there is blurring (integration effect) in the image, the selection of the charge accumulation period (shutter speed) in the image sensor 31 is important. From this point of view, the shutter speed should be as high as possible. However, on the other hand, if the shutter speed is increased, sensitivity is likely to be insufficient. Therefore, it is desirable to compensate the gain with priority on the shutter speed according to the amount of light of the subject. In this embodiment, the shutter speed is set to 1/250 sec (4 ms). When the shutter speed was set to this level for the imaging cycle = 30 ms, no adverse effect due to image blurring in the super-resolution processing was observed visually.

  In the first round motion, as shown in FIG. 14A, imaging is performed at the imaging reference positions P1 to P8. In the second round motion, as shown in FIG. 14B, the imaging reference position P9 is taken. The imaging reference positions P9 to P15 are intermediate positions between the adjacent imaging reference positions (for example, P1 and P2) in the first round motion. When the first and second circular motions are combined, the imaging reference positions P1 to P15 are separated from each other by a relative angle of 24 ° as shown in FIG.

  Here, the first processing mode in which super-resolution processing is performed based on eight images obtained by imaging at the first imaging reference positions P1 to P8 of the circular motion, and the first and second times of the circular motion. And two processing modes, the second processing mode for performing the super-resolution processing, based on the 15 images obtained by imaging at the imaging reference positions P1 to P15.

  In the first processing mode, two imaging reference positions having different positions in both the X-axis and Y-axis directions are set within the original one pixel range, so that the image sensor in each of the X-axis and Y-axis directions. The resolution can be increased at a resolution almost twice the original resolution of 31. On the other hand, in the second processing mode, four imaging reference positions having different positions in both the X-axis and Y-axis directions are set within the original one pixel range. High resolution can be achieved with a resolution almost four times the original resolution of the image sensor 31.

  In particular, in this second processing mode, each of the imaging reference positions P9 to P15 set at the second circular motion is adjacent to the imaging reference positions P1 to P8 set at the first circular motion. Since the imaging reference position is uniformly distributed without being biased, an image excellent in compatibility with the super-resolution processing can be generated.

  It is also possible to perform super-resolution processing in the imaging device 1 while imaging is being performed in the imaging device 1. In this case, in the second processing mode, circular motion is performed twice and 15 What is necessary is just to perform a super-resolution process once whenever the image of a sheet is prepared.

  On the other hand, in the first processing mode, it is preferable to perform super-resolution processing each time eight images are aligned while sequentially shifting the imaging reference position. Specifically, super-resolution processing is performed using the eight images obtained by imaging at the imaging reference positions P1 to P8 at the first time, and at the imaging reference positions P9 to P15 and P1 at the second time. Super-resolution processing is performed using eight images obtained by imaging, and thereafter, imaging reference positions P2 to P9 for the third time, imaging reference positions P10 to P15, P1, and P2 for the fourth time. Are shifted one by one.

  In this way, two processing modes can be set. In both modes, it is not necessary to change the circular motion cycle (the rotational speed of the optical shift mechanism 35) or the imaging cycle, and thus control is easy.

  Note that the first image used for the super-resolution processing in each mode does not need to be limited to the image obtained by imaging at the original position P1, and in the first processing mode, one circular motion is performed from an arbitrary position. Super-resolution processing is performed using 8 images captured during the execution, and in the second processing mode, 15 images captured during two circular motions are performed from an arbitrary position. It may be used to perform super-resolution processing.

  Such processing is performed when super-resolution processing is performed using frame images stored in the storage unit 23 of the image processing device 2 as shown in FIG. The present invention can also be applied to the case where super-resolution processing is performed, and particularly in the latter case, it is not necessary to return the imaging start position to the original position P1 when the processing mode is switched. It is possible to generate high-resolution images with different resolutions by switching.

  As shown in FIG. 2, the imaging cycle is designated by the user using the input unit 26 in the image processing apparatus 2, and the circular motion cycle is determined based on the designated imaging cycle by the cycle setting unit 25. A command signal related to the circular motion period determined here is transmitted to the imaging apparatus 1. The shift control unit 14 of the imaging device 1 operates the optical shift mechanism 35 at a rotational speed corresponding to the designated circular motion cycle based on the command signal regarding the circular motion cycle acquired from the image processing device 2.

  Further, the user can designate the processing mode (first processing mode and second processing mode), and uses the frame image stored in the storage unit 23 of the image processing apparatus 2 as shown in FIG. When performing super-resolution processing, the user is allowed to specify the processing mode together with the reference image, and the number of frame images corresponding to the processing mode specified here is set based on the frame image specified as the reference image. It is only necessary to read out and perform super-resolution processing.

  In addition, the said circular motion period can be changed suitably. For example, by setting the circular motion cycle to 7.2 times the imaging cycle, the imaging reference position returns to the original position at the fifth circular motion, and 36 imagings (sampling) while the circular motion is performed five times. It is also possible to adopt a configuration in which In that case, the respective imaging reference positions are separated by a relative angle of 50 ° (= 360 ° / 7.2).

  Next, with reference to FIGS. 15 to 18, in the image processing apparatus 2 according to the first embodiment, for a predetermined subject, a single high resolution is obtained from a plurality of low resolution images having different imaging positions by optical shift. A super-resolution process for generating an image will be described.

  15 is a functional block diagram of the super-resolution processing unit 24 in the image processing apparatus 2 of FIG. 2, FIG. 16 is a flowchart showing the flow of the super-resolution processing by the super-resolution processing unit 24, and FIG. FIG. 18 is an explanatory diagram illustrating an example of processing of a captured image in super-resolution processing, and FIG. 18 is a block diagram illustrating processing of each frame image in super-resolution processing.

  In FIG. 16, first, the range designation unit 101 (see FIG. 15) sets a range (designated range) designated by the user that requires super-resolution processing in the captured image (S101). The user can input the designated range in advance from the input unit 26 (see FIG. 2).

  Next, the shift cancel unit 102 (see FIG. 15), a series of low-resolution images continuously captured from the storage unit 23, and position information corresponding to each of these images (the coordinates of the image capturing position. As described above, the high (With a finer accuracy than the position of the pixel in the resolution space), and the shift cancellation processing is performed in the low resolution space for the specified range (S102).

  The shift cancel processing means processing for canceling the influence of the optical shift (that is, image movement) applied to the captured image by the optical shift mechanism 35 described above, that is, returning the shift position to the original position. However, since the shift cancellation is performed in units of pixels in the low resolution space as described above, when the optical shift is performed in a range smaller than the pixel size in the low resolution space, the image is shifted before the shift by the shift cancellation processing. It does not return completely to the state.

  Here, for example, as illustrated in FIG. 17, the shift cancel unit 102 performs a captured image having shift position information (0, 0) serving as a reference image to be increased in resolution as a low-resolution image, and the reference image. And a plurality of low-resolution images (here, shifted image information (0, 1) shifted downward from the reference image) and leftward from the reference image. A captured image having shifted shift position information (1, 0) and a captured image having shift position information (1, 1) shifted downward and left from the reference image are acquired. Then, the shift cancel unit 102 performs a shift cancel process for eliminating the optical shift on the basis of the position information for each captured image except the reference image, thereby detecting the captured image (in FIG. 17, a stationary part such as a building). ) Are made to coincide with the position of the reference image (optical shift reference position), respectively. In this case, the shift cancel unit 102 can generate a shift cancel image by shifting the low resolution image in units of a plurality of pixels or one pixel.

  In order to simplify the explanation, FIG. 17 shows an example in which the optical shift is performed vertically and horizontally, but the idea is exactly the same even with the circular motion by the shift mechanism described above. .

  Next, when the motion detection unit 103 (see FIG. 15) acquires the shift cancel image from the shift cancel unit 102, the first detection unit 104 performs still / motion determination on the reference image for each shift cancel image. (S103). Here, the first detection unit 104 detects the pixel block when the average luminance change (difference from the reference image) of the pixel in the target pixel block (processing target region) including one or a plurality of pixels exceeds a predetermined amount. The subject is determined to have moved. The reason for referring to the average value in this way is that the image may not be completely restored to the state before the shift by the shift cancel processing as described above. This is because it includes a plurality of angle vectors, and usually there is almost no case of returning to the state before the shift by the shift cancel process), which causes an erroneous determination.

  If the motion detection unit 103 determines that there is motion (S104; Yes), the second detection unit 105 detects a motion vector in each shift cancel image (S105). Here, as a motion vector detection method, for example, POC (Phase Only Correlation), a block matching method, a gradient method, or the like can be used. Since these are known methods, detailed description thereof is omitted, but for example, Japanese Patent No. 3035654 for POC, Japanese Patent Laid-Open No. 2001-195597 for block matching, and Japanese Patent Publication No. 5-40513 for gradient method. Please refer to each publication.

  Next, when the alignment processing unit 106 (see FIG. 15) obtains motion vector information from the second detection unit 105, the alignment processing unit 106 performs alignment processing for alignment with the reference image (S106). That is, the alignment processing unit 106 corresponds to the corresponding portion of the reference image in the high-resolution image space for the portion that moved in each shift cancel image (in FIG. 17, the portion of the car that runs from left to right in front of the building). A known motion compensation is performed based on the motion vector so as to coincide with the motion vector, and an alignment image for the moved portion is generated. These processes are performed for each pixel block of interest. In FIG. 15, the selection unit 111 sends the alignment image to the reconstruction processing unit 107.

  On the other hand, if it is determined in step S104 that the pixel block of interest is not moving (No), steps S105 and S106 are omitted. In FIG. 15, the selection unit 111 sends the low-resolution image from the storage unit 23 to the reconstruction processing unit 107 instead of the alignment image.

  Thereafter, the reconstruction processing unit 107 (see FIG. 15) performs a reconstruction process using the alignment image of the moving part, the low-resolution image of the stationary part, and the imaging position information, and generates a high-resolution image. (S107). More specifically, the reconstruction processing unit 107 uses the high resolution estimation processing unit 108 to apply each pixel of a plurality of low resolution images to a pixel (subpixel) in the high resolution space by a known method. In particular, when there is no movement in the pixel block of interest, since only the optical shift acts on the pixel block, information on the imaging position of the low-resolution image is known with sub-pixel accuracy in the high-resolution space. The reconstruction processing unit 107 can directly apply the pixels of the pixel block to the high resolution space based on the acquired low resolution image and imaging position information.

  Further, the high-resolution estimation processing unit 108 performs interpolation processing (for example, weighted interpolation according to the distance) for the insufficient region as necessary.

  Next, an inverse filter (Inverse Filter) 109 restores an image using an inverse function of a degradation function generated by a known method.

  The functions shown in FIG. 15 can be realized by hardware, a combination of hardware and software, or software (image processing program). If there is no movement in the pixel block of interest, steps S105 and S106 can be easily omitted, speeding up can be achieved, and calculation cost in the super-resolution processing can be greatly reduced.

  The super-resolution processing as described above is sequentially executed on a plurality (N in this case) of low-resolution images as shown in FIG. FIG. 18 shows a situation in which motion detection and alignment processing are performed in parallel for pixel blocks that occupy the same portion for each of the N low-resolution images. Even when such a configuration is used, When there is no motion in the pixel block to be processed, the processing corresponding to the motion detection unit 103 and the processing corresponding to the alignment processing unit 106 can be easily omitted, the speed is increased, and the calculation cost is reduced. .

(Second Embodiment)
Next, a network camera system according to the second embodiment of the present invention will be described with reference to FIGS. 19 and 20 and FIG. The second embodiment is the same as the case of the first embodiment described above except for the matters specifically mentioned below. In FIG. 19 and FIG. 20, the same components as those in the first embodiment are denoted by the same reference numerals.

  FIG. 19 is a functional block diagram of the super-resolution processing unit 24 according to the second embodiment, and FIG. 20 is an explanatory diagram illustrating an example of captured image processing in the super-resolution processing according to the second embodiment.

  The super-resolution processing in the second embodiment is the same as that in the first embodiment in that a shift cancel image is used instead of a low resolution image in the reconstruction processing in step S107, as indicated by a two-dot chain line in FIG. Not the case. As shown in FIG. 20, the alignment processing unit 106 performs motion compensation based on the motion vector so as to match the corresponding portion of the reference image with respect to a portion where each shift cancel image moves, In step S104 in FIG. 16, if it is determined that the pixel block of interest is not moving (No), the selection unit 111 in FIG. 19 performs shift cancel instead of the alignment image. The shift cancel image from the unit 102 is sent to the reconstruction processing unit 107. In this case, the meaning of the position information is slightly different from that of the low-resolution image in the first embodiment.

  That is, the “position information” of the low-resolution image in the first embodiment means absolute coordinates in the high-resolution space of the captured image, but the “position information” of the shift-cancelled image in the second embodiment is It means relative position information within one pixel in the low resolution space. However, the expression format of “position information” may be the same between the first embodiment and the second embodiment, and only the difference in how “position information” is interpreted in each form.

  As described above, in the first embodiment and the second embodiment, a shift cancel image that cancels the optical shift by the optical shift mechanism in the low resolution space is generated based on the imaging position information, and the motion detection unit If no motion is detected between the reference low-resolution image and the shift-cancellation image, reconstruction processing in the high-resolution space is performed using the low-resolution image or the shift-cancellation image (both have a known imaging position). Is configured to execute.

(Third embodiment)
Next, a network camera system according to a third embodiment of the present invention will be described with reference to FIGS. The third embodiment is the same as the case of the above-described first embodiment except for the matters specifically mentioned below. 21 to 23, the same components as those in the first embodiment are denoted by the same reference numerals.

  FIGS. 21A and 21B are explanatory views showing the positional deviation of the imaging position caused by the relative circular movement of the pixel with respect to the optical image, and FIG. 22 is a functional block diagram of the super-resolution processing unit 24 according to the third embodiment. FIG. 23 is a flowchart showing the flow of super-resolution processing by the super-resolution processing unit 24 according to the third embodiment.

  As described above with reference to FIGS. 13 and 14, each imaging reference position is ideally located on a circular orbit having a predetermined shift radius. For example, FIG. 21a shows an imaging position obtained by adjusting the rotation speed of the shift mechanism so that 64 images are taken per round of the shift mechanism at a shift radius of about 3.8 pixels and 30 frames / second as the optical shift mechanism 35. Shows the trajectory. Even if the optical shift mechanism 35 is continuously driven, the coordinates of the imaging position are not changed at all, and high repeatability is provided. It is possible to actually obtain the characteristics shown in FIG. 21a by designing and creating an optical shift mechanism by appropriately managing parts accuracy, and further optimizing the feedback control variables and the like. As described in the first and second embodiments, the alignment process (steps S105 and S106 in FIG. 16) is omitted based on the information on the imaging position obtained from the optical shift mechanism 35. can do.

  However, for example, when the optical shift mechanism 35 is produced at a low cost, the locus of the imaging position (indicated by a ♦ in the drawing) of the optical shift mechanism 35 during rotation is shown in FIG. 21b. A positional shift (optical shift error) may occur due to vibration of the component, variation in the output of the radial magnetic sensor 39, change with time, or the like. For example, in FIG. 21b, when the optical member 51 is rotated by an angle θ from the origin sensor reference position Po, the shift control unit 14 outputs the position Pp on the circular orbit as the imaging position, but the imaging actually determined from the image is performed. The position is the position Pr.

  When the inventors diligently investigated such an actual imaging position error, it is clear that the imaging position (imaging reference position) output from the shift control unit 14 and the imaging position actually determined from the image are clear. It was found that the generated error is within a predetermined range in the high resolution space. In other words, even when a positional deviation of the imaging position occurs, the amount of calculation can be reduced by performing the alignment process in which the search range is limited to the range of error that occurs (here, the range W of one pixel in the high resolution space). It has been found that it can be greatly reduced.

  In FIG. 23, when the motion detection unit 103 (see FIG. 22) determines that there is motion (S104; Yes), the second detection unit 105 causes the motion vector within a predetermined search range (first range) for each shift cancel image. Is detected (S105), and the first alignment processing unit 106 (see FIG. 22) performs alignment processing in the same manner as in the first embodiment to generate an alignment image (S106).

  On the other hand, when it is determined in step S104 that the pixel block of interest is not moving (No), the third detection unit 121 (see FIG. 22) detects a motion vector based on the position information from the storage unit 23 ( S108). At this time, the search range (second range) of the third detection unit 121 is set within a range of one pixel (sub-pixel) width in the high resolution space, and the search range (for example, high resolution) of the second detection unit 105 is set. Smaller than 100 pixels in space). Then, the second alignment processing unit 122 (see FIG. 22) performs motion compensation based on the motion vector detected by the third detection unit 121, and generates an alignment image for the moved part (S109).

  In this case, when the first detection unit 104 determines that the first detection unit 104 has motion (S104; Yes), the selection unit 111 re-registers the alignment image that has been subjected to motion compensation by the first alignment processing unit 106. On the other hand, if it is determined that there is no motion (S104; No), the alignment image subjected to motion compensation by the second alignment processing unit 122 is output to the reconstruction processing unit 107.

  Although this invention was demonstrated based on specific embodiment containing an Example, these embodiment is an illustration to the last and this invention is not limited by these embodiment. Note that all the components of the imaging apparatus according to the present invention shown in the above-described embodiments are not necessarily essential, and can be appropriately selected as long as they do not depart from the scope of the present invention.

  An imaging system and an image processing apparatus, an image processing method, and an image processing program used for the imaging system according to the present invention can reduce calculation cost by suppressing unnecessary registration processing in super-resolution processing. A high-resolution image is generated by super-resolution processing from a plurality of low-resolution images acquired by so-called pixel shift, which performs imaging while relatively displacing the optical image formed on the light-receiving surface of the element and the image sensor. And an image processing apparatus, an image processing method, and an image processing program used therefor.

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Image processing apparatus 11 Imaging part 12 Image processing part 14 Shift control part (position information acquisition part)
31 Image sensor 35 Optical shift mechanism 36 Magnetic rotation drive part 38 Position control part 39 Radial direction magnetic sensor 40 Axial direction magnetic sensor 41 Sensor module 42 Lens unit (optical system)
50 Optical capsule 51 Optical member 52 Liquid (fluid)
53 Parallel plate 54 Back yoke 55 Diameter side magnet 56 Upper magnet 57 Lower magnet 65 Origin sensor 72 Position control coil 75 Electromagnet 76 Permanent magnet 77 Connecting member 102 Shift canceling unit 103 Motion detecting unit 106 (First) Positioning processing unit 107 Configuration processing unit 122 Second alignment processing unit C Optical axis

Claims (7)

A plurality of low-resolution images captured by causing the image sensor and a light image formed on the light-receiving surface of the image sensor to move relative to each other are acquired, and the plurality of low-resolution images having different imaging positions are acquired. An image processing device for generating a high-resolution image from
A shift cancel unit that generates a shift cancel image based on the information of the imaging position of the low resolution image, the position of the low resolution image matched with the position of the reference low resolution image serving as a reference;
A motion detection unit that detects a motion in the low resolution image by comparing the shift cancel image and the reference low resolution image;
An alignment processing unit that generates an alignment image by aligning the shift cancel image with the reference low-resolution image based on the result of the motion detection;
A reconstruction processing unit that generates a high-resolution image by reconstruction processing,
The reconstruction processing unit executes the reconstruction process using the alignment image when the motion detection unit detects a motion for a predetermined processing target area of the shift cancel image, An image processing apparatus that executes the reconstruction processing using the low-resolution image or the shift-cancelled image when no motion is detected by a motion detection unit.   The image processing apparatus according to claim 1, further comprising a range specifying unit that specifies a range that can be the processing target region in the low-resolution image. The motion detection unit includes a first detection unit that determines the presence or absence of motion for the shift cancel image, and a second detection unit that detects a motion vector for the shift cancel image,
The image processing apparatus according to claim 1, wherein the alignment processing unit performs the alignment based on the motion vector.   The image processing apparatus according to claim 1, wherein the shift cancel unit generates the shift cancel image by shifting the low-resolution image in units of pixels. An imaging system comprising: the image processing apparatus according to any one of claims 1 to 4; and an imaging apparatus that captures the low-resolution image.
The imaging device
An image sensor that photoelectrically converts light from a subject and outputs a pixel signal;
An optical system for guiding light from the subject to the image sensor;
An optical shift mechanism for performing a relative circular motion between the optical image formed on the light receiving surface of the image sensor and the image sensor;
An image pickup system comprising: a position information acquisition unit that acquires information on the image pickup position of the image pickup element. A plurality of low-resolution images captured by causing the image sensor and a light image formed on the light-receiving surface of the image sensor to move relative to each other are acquired, and the plurality of low-resolution images having different imaging positions are acquired. An image processing method for generating a high resolution image from
A shift cancel generation step of generating a shift cancel image in which the position of the low resolution image is matched with the position of the reference low resolution image based on the information of the imaging position of the low resolution image;
A motion detection step of detecting a motion in the low resolution image by comparing the shift cancel image and the reference low resolution image;
An alignment processing step of generating an alignment image by aligning the shift cancel image with the reference low resolution image based on the result of the motion detection;
A reconstruction processing step for generating a high-resolution image by reconstruction processing,
In the reconstruction processing step, when a motion is detected in the motion detection step for a predetermined processing target region of the shift cancel image, the reconstruction processing is performed using the alignment image, An image processing method, wherein when no motion is detected in the motion detection step, the reconstruction processing is executed using the low resolution image or the shift cancel image.   An image processing program for executing the image processing method according to claim 6 by controlling the image processing apparatus.

Images