JP5897309B2 - Image processing method and apparatus - Google Patents

Description

The present invention relates to an image processing method and apparatus, and more particularly to the field of imaging as a measurement technique for imaging and visualizing target information (physical information such as position, brightness, color, and magnetization). Is.

Special optical systems and detectors need to be combined in order to capture weak signal changes, even on signals from minute structures such as nanoscale and microscale, and even on a macroscale. For example, graphene, which is a monolayer carbon film, has a micrometer size scale and low reflectivity, so by combining a high magnification, high precision optical microscope, CCD camera, special filter, etc. Observation is possible for the first time.

That is, the so-called imaging technique that makes it possible to see what is normally invisible is generally in a situation where it is necessary to rely on expensive and special devices. This has many problems in terms of economy, and there are also problems in flexibility such as incorporation into existing devices, and versatility when used in new fields.

The present invention intends to cope with such a problem by using a plurality of images acquired by shifting the relative positions of the measurement target and the measurement system. Conventionally, as represented by camera shake correction of a digital camera, there has been a fixed idea that an object or a measurement system is firmly fixed in order to accurately measure the object. The present invention is based on the idea opposite to the conventional fixed idea of acquiring an image by shifting the relative position between a measurement object and a measurement system.

Obtaining a shifted image by moving the object and the measurement system of the image acquisition device itself is also performed in, for example, a so-called pixel shifting method (Patent Documents 1 to 3). The pixel shifting method uses pixel sensors to improve the resolution that is originally affected by the size and arrangement of cells when acquiring images using an image sensor with light receiving cells, such as a CCD camera or CMOS camera. For example, it is a technique in which images are taken while being shifted by half a pixel, and the acquired image is added. The fact that the present invention and the pixel shifting method are different technical ideas will become apparent by reading this specification.

In image analysis, edge detection using a differential filter is known. As a differential filter, a Sobel filter is known as a primary differential (difference) filter, and a Laplacian filter is known as a secondary differential (difference) filter. The Sobel filter is based on one image and the pixel values of the neighboring pixels (2 directions and 6 directions) of the pixel of interest. The Laplacian filter is the neighboring pixel (4 directions and 8 directions) of the pixel of interest based on one image. This is a numerical calculation using each pixel value. Even in one aspect of the present invention, the contour of the object can be detected. However, the present invention and the edge detection using the differential filter are different from each other in the technical idea. Will become clear.

In general, a scanning microscope is known as a microscope having a high spatial resolution, and an optical microscope includes a scanning microscope (Patent Document 4). In a scanning optical microscope, for example, a two-dimensional image is acquired from two-dimensional mapping data obtained by moving a sample stage in an XY plane with respect to an irradiation position of a spot light. Scanning is performed in order to acquire one two-dimensional image. In one aspect of the present invention, a sample stage that can move in the XY plane is used. However, the movement of the sample stage in the present invention is different from the movement of the sample stage in the scanning microscope in terms of technical idea. It becomes clear by reading this specification.

JP 7-212660 A JP-A-8-95153 JP 2001-251491 A JP 2003-255231 A

As described above, in order to obtain weak image information that is usually difficult to observe, the optical system and the detector must be special and expensive. This gives rise to cost problems and hinders application to new observation systems.
It is an object of the present invention to provide an imaging technique that can improve spatial resolution with a simple configuration and is versatile at low cost.

The technical means adopted by the present invention to solve this problem are as follows:
An image processing method using an image acquisition device for acquiring a target image,
Obtaining a target image as a base image;
Obtaining N shift images (N ≧ 1) obtained by shifting the object by a predetermined amount in the imaging plane with respect to the base image;
Obtaining N difference images (N ≧ 1) from the difference between pixels of the same coordinates of the base image and each shift image;
Adding the N difference images to obtain a target difference integrated image;
An image processing method comprising:

In one aspect, the difference accumulated image is obtained by subtracting a base image from each shift image and accumulating.
In one aspect, the image processing method further includes a step of subtracting the difference integrated image from a base image to obtain a target sharpened image.
When acquiring a sharpened image, the base image may be weighted. For example, the weight is selected from a range of 1 to 20 times.

A difference integrated image obtained by subtracting the base image from the shift image and integrating is a secondary differential image. Then, by subtracting the secondary differential image from the base image, the high intensity is stronger, the weak intensity is weaker, the contour is emphasized, and a sharpened image is obtained.
In addition, when only the difference image is used, if only the outline of a certain target is extracted, the sign of the value of each pixel of the difference image does not matter (the difference between emphasizing with white or emphasizing with black) It will be understood by those skilled in the art that the sign inversion of the difference image can be performed. Similarly, those skilled in the art will understand that sign inversion of a sharpened image can be appropriately performed. It is understood by those skilled in the art that “adding” and “subtracting” can be replaced in accordance with the sign inversion in the description of the present specification and claims.

In one aspect, the predetermined amount is an amount in the range of 1/10 to 2 times the spatial resolution of the image acquisition device.
In one aspect, the predetermined amount is an amount in the range of 1/5 to 1 times the spatial resolution of the image acquisition device.
The spatial resolution of the image acquisition device is an important specification, and can be estimated from known or the configuration of the device. For example, in the optical fiber imaging apparatus used in the experiment, it is considered that the fiber resolution of 2 to 3 μm is the spatial resolution, and in the Nomarski differential interference optical microscope (BX60) used in the experiment, it can be estimated to be approximately 1 μm. it can.

In one aspect, the shift image is acquired by moving a target or / and a measurement system of an image acquisition device.
The movement of the object is typically to move the object on a moving mechanism (for example, a piezo drive stage, a belt conveyor) by the moving mechanism, but the object itself has mechanical moving means and vibration means. In this case, the case where a subject (typically a living organism) moves by itself is included.
In the case of moving the measurement system of the image acquisition device, the case of moving the entire measurement system and the case of moving a part of the measurement system are included. For example, when the measurement system of the optical image acquisition apparatus includes an optical system and a detector, when moving the “optical system + detector”, the “detector” may be moved.
In one aspect, the N shifted images are eight images shifted in two directions in the x direction, two directions in the y direction, and four directions in the oblique direction with respect to the base image.
In one aspect, the N shifted images are four images shifted in two directions in the x direction and two directions in the y direction with respect to the base image.
In one aspect, the N shifted images are two images shifted in two directions, the x direction and the y direction, with respect to the base image.
The shift direction is not limited to a pair of opposite directions (such as two directions in the x direction), and may be an odd direction including one direction, and also includes rotation.

The present invention is also realized as an image processing device using an image acquisition device that acquires a target image.
Means for acquiring a target image as a base image;
Means for acquiring N shift images (N ≧ 1) in which the object is shifted by a predetermined amount within the imaging plane with respect to the base image;
An image processing unit,
The image processing unit acquires N difference images (N ≧ 1) from a difference between pixels having the same coordinates in the base image and each shift image, and adds the N difference images to add a difference of a target. Get an image.

  In one aspect, the image processing unit acquires a difference integrated image by subtracting and integrating a base image from each shift image, and acquires a target sharpened image by subtracting the difference integrated image from the base image. To do.

In the present invention, the image acquisition device for acquiring the target image may be any device as long as it can acquire the target base image and shift image. Specifically, an optical microscope (stereoscopic microscope, phase contrast microscope, differential interference microscope, (Including polarization microscope, fluorescence microscope), transmission electron microscope, ultrasonic microscope, digital camera, fiberscope, thermography, magnetic resonance (MR) imaging device, ultrasonic imaging device, optical ultrasonic imaging device, Raman spectroscopic device, microscope Examples include Raman spectroscopic devices and radiation imaging devices (including X-rays, alpha rays, beta rays, gamma rays, and neutron rays).
In the embodiments to be described later, experiments were performed using two types of image acquisition devices, an optical fiber imaging device and a Nomarski differential interference optical microscope. It will be understood by those skilled in the art that this technique can be appropriately executed by those skilled in the art to apply to other image acquisition apparatuses other than the optical fiber imaging apparatus and the Nomarski differential interference optical microscope.
In the present invention, the shift direction of the measurement system of the object or / and the image acquisition device is a direction parallel to the imaging plane. That is, when moving the object, the shift direction is perpendicular to the optical axis in an optical image acquisition device, and perpendicular to the beam axis in an image acquisition device using an electron beam or radiation. .

In the present invention, a shift image is acquired by shifting the target position relative to the measurement system, and a difference image between the base image and the shift image is acquired (the base image is subtracted from the shift image). With a simple operation, a significant part of noise other than the signal to be detected (temporal and spatial fluctuations in the measurement system, more specifically, fluctuations in the light source, fluctuations in the optical system, fluctuations in the detector, etc.) are removed. Thus, only the sample position dependency of the signal intensity can be detected as a difference image with high sensitivity. The difference integrated image formed by adding the obtained difference images is a target spatial second-order differential image, and an image with improved spatial resolution can be obtained by subtracting the second-order spatial differential image from the base image. it can.

Spatial resolution can be improved by a simple method of photographing and calculating while shifting the position of the sample stage of an existing imaging apparatus (various microscopes, cameras, inspection apparatuses, etc.). For devices that already have a movable stage, it is only necessary to change the imaging process and calculation method, and even when the present invention is applied to an existing device that does not have a movable stage, a movable stage is added. The imaging apparatus can be realized at low cost.

The effects of the present invention are not limited to the above description, and other effects will become apparent from the description of the present specification.

It is a block diagram which shows embodiment of this invention. It is a figure explaining the concept of this invention. It is a figure which shows acquisition of a shift image (spatial secondary differential imaging process). A base image C ₀ and four shifted images C ₁ to 4 shifted in four directions are shown. The noise component is zero. The image is a white (pixel value 1) cross with a gray scale mark (pixel value 0.9) in the center, and the shift amount is 1 pixel. It is the figure which displayed FIG. 3 by the pixel value. It is a figure which shows the example of an imaging process (in the case of 1 direction), an upper figure is a base image, a center figure is a difference image, and a lower figure is a sharpened image. It is a figure which shows the example of an imaging process (in the case of 2 directions), an upper figure is a base image, a center figure is a difference integration image, and a lower figure is a sharpened image. It is a figure which shows the example of an imaging process (in the case of 4 directions), an upper figure is a base image, a center figure is a difference integration image, and a lower figure is a sharpened image. It is a figure which shows the effect of the weight of the base image in a sharpened image. It is a figure which shows acquisition (in the case of 4 directions) of the shift image in case a shift amount is 1/5 pixel. A base image C ₀ and four shifted images C ₁ to 4 shifted in four directions are shown. The noise component is zero. It is a figure which shows the example of an imaging process (in the case of 4 directions) in case shift amount is 1/5 pixel, an upper figure is a base image, a center figure is a difference integration image, and a lower figure is a sharpened image. It is a figure which shows the effect of the weight of the base image in a sharpened image. The shift amount is 1/5 pixel. It is a figure which shows the optical fiber imaging device used for experiment. It is a figure which shows acquisition of the shift image of 8 directions. It is a figure which shows the shift amount dependence (shift amount: 0 micrometer, 0.25 micrometer, 0.5 micrometer) of a difference integration image and a sharpened image. It is a figure which shows the shift amount dependence (shift amount: 1 micrometer, 2 micrometers, 3 micrometers) of a difference integration image and a sharpened image. It is a figure which shows the shift amount dependence (shift amount: 4 micrometers, 5 micrometers) of a difference integration image and a sharpened image. It is a figure which shows the observation result of a graphene hole bar. It is a figure which shows the observation result of single layer graphene / graphite. It is a figure which shows the observation result of single layer / bilayer graphene. It is a figure which shows the observation result of a single layer / multilayer graphene. It is a figure which shows the observation result of a single layer / multilayer graphene. It is a figure which shows the observation result of a single layer / multilayer graphene. It is a figure which shows the observation result of a single layer / multilayer graphene. It is a figure which shows the effect (observation result) of the weight of the base image in a sharpened image. It is a figure which shows the effect (observation result) of the weight of the base image in a sharpened image. It is a figure which shows the effect (observation result) of the weight of the base image in a sharpened image. It is a figure which shows the effect (observation result) of the weight of the base image in a sharpened image. It is a figure which shows the observation object (what was acquired by Olympus Microscope BX-60). It is a figure which shows embodiment (application 1) as an inspection apparatus using a belt conveyor. It is a figure which shows embodiment (application 2) as an inspection apparatus using a belt conveyor. It is a figure which shows embodiment (application 3) as an inspection apparatus using a belt conveyor.

[A] Imaging device
[A-1] Overview of Image Processing in Imaging Apparatus FIG. 1 shows a block diagram of an embodiment of an image processing apparatus (imaging apparatus) according to the present invention. The image processing apparatus includes an image acquisition unit (measurement system) that acquires an image of a target, and a unit that moves the target relative to the measurement system. In the following description, an embodiment of the present invention will be described based on an optical image acquisition device, but an image acquisition device that can be applied to the present invention is not limited to an optical image acquisition device.

A base image and a shift image are acquired by the image acquisition unit and the moving means. That is, the base image is moved by moving the target and / or the measurement system of the image acquisition device by fixing the target position and acquiring the target image as a base image from the image acquisition unit and the moving unit. Means for acquiring N shift images (N ≧ 1) in which the target is shifted by a predetermined amount in the imaging plane.

The image processing apparatus includes an image processing unit. The image processing unit obtains N difference images (N ≧ 1) from the difference in pixel value (signal intensity) between pixels having the same coordinates in the base image and each shift image, and the same coordinates in the N difference images. The pixel value (signal intensity) between the pixels is added to obtain the target difference integrated image. When N = 1, difference image = difference integration image. The image processing unit obtains a difference accumulated image by subtracting and integrating the base image from each shift image, and obtains a target sharpened image by subtracting the difference accumulated image from the base image.

The image processing apparatus includes a display unit, and an acquired image (that is, a base image, a shift image), data calculated using the acquired image, and the like can be appropriately displayed on the display unit.

The image processing unit can be configured from a general-purpose computer (including an input unit, an output unit, a calculation unit, a storage unit, a display unit, and the like). The acquired image data and calculation data are stored in the storage unit, and various calculations are executed by the calculation unit. The display unit of the image processing apparatus can be configured from a display unit of a general-purpose computer.

In the image processing method according to the present embodiment, the target position is fixed, the target image is acquired as a base image, and the measurement system of the target or / and the image acquisition device is moved to the base image. On the other hand, obtaining one or a plurality of shift images in which the target is shifted by a predetermined amount in the imaging plane, and one or a plurality of difference images from the difference between pixels of the same coordinates of the base image and each shift image And a step of adding pixels having the same coordinates of the one or more difference images to obtain a target difference integrated image.

The image acquired in each step only needs to be acquired as a numerical value as image data, and is not necessarily displayed on the display device. For example, only the finally obtained difference integrated image may be displayed and other images may not be displayed.

Each step does not necessarily specify a time series, and some steps may be executed simultaneously. For example, the step of acquiring a plurality of difference images and the step of acquiring a target difference integrated image may be performed simultaneously. You may perform simultaneously the step which acquires a some difference image, the step which acquires the difference integrated image of object, and the step which adds a base image to a difference integrated image. Alternatively, the shift image may be acquired before acquiring the base image in time series. For example, an object moving in one direction may be taken continuously, and in a set of three consecutive images, the image _t may be a base image, and the images _t−1 and _{t + 1} may be shifted images. About this, the application 1 and 2 to the test | inspection apparatus using the belt conveyor mentioned later can be referred.

A conceptual diagram of the present invention is shown in FIG. As shown in FIG. 2, the base image consists of a true signal and noise that does not depend on the sample position. The shifted image also includes noise that does not depend on the sample position. If the shift image is composed of a first shift image shifted to the left with respect to the base image and a second shift image shifted to the right with respect to the base image, a difference image obtained by subtracting the first shift image from the base image And a difference image obtained by subtracting the second shift image from the base image is added to obtain a difference integrated image. The difference integrated image is a spatial second derivative mapping of the light intensity in the shift direction. The difference integration image is information on the contour of the target.

The feature of this embodiment is that only the sample position dependence of the signal intensity can be extracted by shifting the entire measurement system including the optical system and the detector by a small amount relative to the sample (difference method). The difference integrated image generated by this operation is a spatial second-order differential image. In the imaging apparatus, if the position of the object is shifted and photographed, and calculation (addition and subtraction) between images is performed, an observation apparatus based on the difference method is obtained. Here, if the object is shifted by a distance corresponding to the spatial resolution of the apparatus, a considerable part of noise other than the signal to be detected (light source fluctuation, optical system fluctuation, detector fluctuation, etc.) is removed. And the spatial resolution of the apparatus can be improved. That is, if “signal fluctuation (noise)” = “light source fluctuation” + “optical system fluctuation” + “scanning fluctuation” + “detector fluctuation” + “other fluctuations”, there is no change due to the position shift. Noise can be almost canceled.

The processing by the differential filter is shown for comparison on the lower side of FIG. Whereas the differential filter is a numerical operation using the pixel value of the pixel near the pixel of interest, in the present invention, an optical system for the object, a measurement system including a detector, or a detector for imaging Is shifted by a predetermined amount, and pixel values between pixels having the same coordinates in a plurality of actually acquired images are used, and they are completely different. It is clear from the experimental results described later that the results obtained by these methods are completely different.

Typically, the image processing method further includes a step of subtracting the difference integrated image from a base image to obtain a target sharpened image. That is, the base image is added to the difference accumulated image. In this case, the base image may be weighted (for example, 1 to 20 times). When the base image is subtracted from the shift image and integrated, the difference integrated image becomes a second-order differential image. Then, by subtracting the second-order differential image from the base image, the high intensity is stronger, the weak intensity is weaker, the contour is emphasized, and a sharpened image is obtained. It should be understood by those skilled in the art that when only the difference image is used, the sign of the difference image can be reversed if only the contour of a certain target is extracted. Similarly, those skilled in the art will understand that sign inversion of a sharpened image can be appropriately performed. The second-order differential image is mathematically obtained by subtracting the base image from the shift image, and in that sense, the difference accumulated image can be defined as the sum of the shift image minus the base image. The sharpened image is obtained by subtracting this difference integrated image from the base image. According to this definition, in the difference integrated image, the bright side of the contour is dark and the dark side is bright, resulting in an unnatural impression. In the difference accumulated image of actual experimental data, the sign is inverted so that the bright side is brighter and the dark side is darker for easy viewing.

[A-2] The image subtraction performed by the arithmetic image processing unit in the image processing unit will be described. In the present embodiment, the addition of a plurality of images is an addition of pixel values (signal strength) of pixels having the same coordinates in each image. Assuming that the image before moving the stage on which the sample is placed (that is, the image to be acquired) is the base image ₀ , and the images acquired when the stage is moved in the eight directions of up, down, left, and right are the shift images ₁ to ₈ , the difference The calculation of the integrated image is a subtraction of the two-dimensional image. Expressed by the equation, (shift image ₁ + shift image ₂ + shift image ₃ + shift image ₄ + shift image ₅ + shift image ₆ + shift image ₇ + shift image ₈ ) −base image ₀ × 8. For example, when focusing on the pixel (M5, N5), {the value of the pixel (M5, N5) of the shift image ₁ + the value of the pixel (M5, N5) of the shift image ₂ + the pixel (M5, N5) of the shift image ₃ Value + value of pixel (M5, N5) of shift image ₄ + value of pixel (M5, N5) of shift image ₅ + value of pixel (M5, N5) of shift image ₆ + pixel of shift image ₇ (M5, N5) ) Value + shifted image ₈ pixel (M5, N5) value} −base image ₀ pixel (M5, N5) value × 8. As described above, the numerical calculation (Laplacian filter) is a calculation between adjacent pixels, whereas in the present embodiment, the calculation is performed between pixels at the same position (coordinates) on a plurality of images.

Processing executed by the image processing unit will be described based on the image data shown in FIGS. FIG. 3 is a diagram showing acquisition of a shift image (spatial second-order differential imaging process), and shows a base image C ₀ and four shift images C ₁ to 4 shifted in four directions. The noise component is zero. The image is a white (pixel value 1) cross having a gray scale mark (pixel value 0.9) in the center, and the pixel values around the cross are 0. In order to simplify the processing, the shift amount is set to one pixel for convenience. FIG. 4 is a diagram in which FIG. 3 is displayed with pixel values. By comparing the pixel values of the coordinates of the base image C ₀ and the four shift images C _{1 to 4} , the shift image C ₁ to ₄ can shift the base image C ₀ leftward, rightward, upward, and downward, respectively. It can be read that the image is shifted by one pixel. Assuming that the base image C ₀ is in the XY plane, the shifted images C ₁ and C ₂ can be referred to as images shifted in two directions in the X direction, and the shifted images C ₃ and C ₄ can be referred to as images shifted in two directions in the Y direction. .

FIG. 5 is a diagram illustrating an example of an imaging process (in the case of one direction), in which the upper diagram is a base image, the center diagram is a difference image, and the lower diagram is a sharpened image. Shift image is shifted image C ₁ obtained by one pixel shifted base image C ₀ in the left direction. Difference image is obtained by the shift image C ₁ subtracts the base image C _0. Sharpened image can be obtained by the base image C ₀ is subtracted the difference image.

FIG. 6 is a diagram illustrating an example of an imaging process (in the case of two directions), in which the upper diagram is a base image, the central diagram is a difference integrated image, and the lower diagram is a sharpened image. Shift image base image C ₀ shift image C ₁ obtained by one pixel shifted _leftward, a shift image C ₂ obtained by one pixel shifted base image C ₀ in the right direction. The difference integrated image is obtained by adding the shift images C ₁ and C ₂ and subtracting the base image C ₀ × 2. Sharpened image can be obtained by the base image C ₀ is subtracted the difference accumulated image.

FIG. 7 is a diagram illustrating an example of an imaging process (in the case of four directions), in which the upper diagram is a base image, the central diagram is a difference integrated image, and the lower diagram is a sharpened image. Shift image base image C ₀ leftward, rightward, upward, and shift the image C _{1 to 4} comprising by each one pixel shifted downward. The difference accumulated image is obtained by adding the shift images C ₁ , C ₂ , C ₃ , C ₄ and subtracting the base image C ₀ × 4. Sharpened image can be obtained by the base image C ₀ is subtracted the difference accumulated image.

Figure 8 is a diagram showing the effect of the weight of the base image C ₀ at the acquisition of the sharpened image. The weights are -1, -0.5, 1, 10, and 20 in order from the left.

When the shift amount is not an integer multiple of the pixel, the signal intensity is divided by the area ratio. FIG. 9 is a diagram illustrating acquisition of a shift image (in the case of four directions) when the shift amount is 1/5 pixel. A base image C ₀ and four shifted images C ₁ to 4 shifted in four directions are shown. The noise component is zero.

FIG. 10 is a diagram showing an imaging process example (in the case of four directions) when the shift amount is 1/5 pixel. The upper diagram is a base image, the central diagram is a difference integrated image, and the lower diagram is a sharpened image. Shift image base image C ₀ leftward, rightward, upward, and shift the image C _{1 to 4} comprising by each 1/5 pixel shifted downward. The difference accumulated image is obtained by adding the shift images C ₁ , C ₂ , C ₃ , C ₄ and subtracting the base image C ₀ × 4. Sharpened image can be obtained by the base image C ₀ is subtracted the difference accumulated image. Although the absolute value of the value of each pixel of the difference integrated image and the sharpened image is reduced by the amount of shift, the same image pattern as in the case of shift amount = 1 was obtained.

FIG. 11 is a diagram illustrating the effect of the weight of the base image in the sharpened image. The shift amount is 1/5 pixel. The weights are -1, -0.5, 1, 10, and 20 in order from the left. Compared to the shift amount = 1, when the weight is applied to the base image, the contour tends to be inconspicuous.

[A-3] As a means for moving the movement target of the measurement system of the target / image acquisition apparatus, a piezo drive stage using a piezo element can be typically exemplified. Such a piezo-drive stage is widely used in an optical microscope for the purpose of searching for a sample or moving it to an easily viewable position. The piezo drive stage is also used as a scanning stage for acquiring a two-dimensional image in a scanning microscope as exemplified in Patent Document 4. However, the use of these piezo drive stages has a different technical idea from the movement for processing a plurality of shifted images and using them for image sharpening in this embodiment.

Moving the measurement system of the object and the image acquisition device itself is also performed in Patent Documents 1 to 3, and when acquiring the shift image in the present invention, the methods disclosed in these documents are adopted. Also good. However, the pixel shifting method, when acquiring an image using an image sensor in which cells are spread, such as a CCD camera or a CMOS camera, originally improves the spatial resolution that is influenced by the size and arrangement of the light receiving cells. Note that this is a technique in which pixel positions are shifted by, for example, half a pixel and an acquired image is added, and therefore noise is also added, which is different from the present invention. Patent Documents 2 and 3 also disclose a piezo drive stage. However, the pixel shift methods disclosed in Patent Documents 2 and 3 add the acquired images, and acquire the difference image in the present invention. However, it does not have a configuration for acquiring a difference integrated image, and both are based on completely different technical ideas. The present invention differs from the pixel shifting method in that the former is based on spatial resolution as the shift amount, whereas the latter is based on the pixel size. The pixel shifting method is effectively applied to an image acquisition device having an image sensor in which light receiving cells are spread, such as a CCD camera or a CMOS camera. For example, the spatial resolution is determined by the diameter of the fiber strand. There is no significance in applying the pixel shifting method to optical fiber imaging devices or optical devices determined by lens aberration and light wavelength. When the spatial resolution of the measurement system is determined by the pixel size, as a result, the movement amount in the present invention and the pixel shift method can overlap, but the overall configuration of both is different as described above.

Instead of or in addition to moving the object, the measurement system of the image acquisition device may be moved. For example, an imaging fiber whose tip is periodically oscillated and an image may be acquired in synchronization therewith. Moving the measurement system of the image acquisition apparatus includes moving the entire measurement system and moving only some elements of the measurement system. More specifically, when the measurement system includes an optical system and a detector, the detector is finely shifted when the entire measurement system is shifted by a small amount relative to the object, or for image formation by the optical system. This includes the case of a small shift.

The movement of the measurement system (optical system + detector) of the object or / and the image acquisition apparatus will be further described. When the image acquisition device is regarded as an object + optical system + detector, first, regarding “object” + “optical system + detector” in which the optical system and the detector are fixed, the shift is relative, The effect is the same regardless of whether the “object” or “optical system + detector” is shifted. In addition, with respect to “object + optical system” + “detector” in which the object and the optical system are fixed, the same effect can be obtained by shifting only “detector” or “object + optical system”. can get.

Here, it should be noted that the noise that can be removed differs between when the optical system and the detector are fixed and when the target and the optical system are fixed. The former can mainly remove noise due to the optical system and the detector, whereas the latter can mainly eliminate only noise due to the detector. For example, when a digital camera is used as an image acquisition device, it is conceivable to use the latter method, and the present invention can be realized by a relative position shift between “object + optical system” and “detector”. If the optical system is fixed, the image formed by the target optical system can be regarded as a new target and transferred to the detector. A device that continuously captures images while shifting the image sensor in the camera by a small amount, and records / displays the images by performing internal calculation is conceivable. Since it can be considered that the normal object is not moving with respect to the shutter speed, only the image sensor of the measurement system is moved. At this time, only the CCD or CMOS image sensor moves with respect to the target and the lens system inside the camera. The shift amount is about the spatial resolution, that is, the cell size of the image sensor. It is thought that sensitivity variations due to non-uniformity of CCD or CMOS image sensor elements, which account for a lot of noise in digital cameras, can be improved.

[A-4] It is considered that the shift amount of the shift image relative to the distance base image between the base image and the shift image is preferably about the spatial resolution of the image acquisition device. However, the definition of spatial resolution is ambiguous, the distance between two points that can be identified, and the judgment of “visible” or “invisible” is made by calculating the level of the brain, and how much is the contour as an image Whether the emphasis is easier to see or the texture of the object is not impaired depends on the individual on the viewing side, and also differs depending on the object that is the image. Therefore, the amount of movement that can be employed in this embodiment has a range, and in one aspect, the amount of movement of the shift image relative to the base image is 1/10 to 2 times the spatial resolution of the image acquisition device. In a more preferable range, it is 1/5 to 1 times the spatial resolution. This range of movement amount is a range supported by the experimental results described later. In fact, the edge information is obtained although it is weak at about 1/10 of the spatial resolution. On the other hand, the amount of movement includes a value larger than the spatial resolution, and if it is several times the spatial resolution, for example, about twice, it may be seen more clearly than the base image.

The spatial resolution of the image acquisition device is an important specification, and can be estimated from known or the configuration of the device. For example, in the optical fiber imaging apparatus used in the experiment, it is considered that the diameter of the fiber strand of 2 to 3 μm is the spatial resolution, and in the Nomarski differential interference optical microscope (BX60) used in the experiment, it can be estimated as approximately 1 μm. it can. Accordingly, a person skilled in the art sets an appropriate shift amount (typically about spatial resolution) from a range of 1/10 to 2 times the spatial resolution based on the spatial resolution of the image acquisition device used. Can do.

Typically, the movement amount is set to substantially the same amount in each direction (as much as possible to minimize the error). For example, in four shift images in four directions, the amount of movement of each shift image with respect to the base image is substantially the same. In general, the imaging apparatus is used not for quantitative analysis but for judging whether it can be seen or not. Therefore, even if there is variation in the movement amount between the shift images, the movement amount is within the predetermined range. If it is in, there is an effect. When the movement amount is different, the difference image may be weighted according to the movement amount in order to match the contour strength.

[A-5] When obtaining a weighted difference integrated image of an image, only a specific difference image may be added or weighted. For example, when there are ₈ shift images _{1 to} 8 in the eight directions of the base image ₀ and the top / bottom / left / right slant, only a part of the eight difference images may be selected and added (in this case, selected) The weight of the difference image that did not exist is 0). In addition, for example, in the case of a four-direction shift image, the difference image may be weighted when the shift amount differs vertically and horizontally. Strictly speaking, by setting (image ₁ −image ₀ ) / (shift amount of image ₁ ) + (image ₂ −image ₀ ) / (shift amount of image ₂ ) +. Can be made the same. Further, the weight may be changed in a special application, for example, when it is desired to emphasize only the vertical contour. How to apply the weight can be appropriately set by those skilled in the art according to the purpose and application.

When the base image is added to the difference accumulated image to obtain the target sharpened image, the base image may be weighted. How to perform weighting can be appropriately selected by those skilled in the art from the range of 1 to 20 times, for example, depending on the purpose and application.

[A-6] Movement direction of shift image with respect to base image In one aspect, the plurality of shift images are shifted in two directions in the x direction, two directions in the y direction, and four directions in the oblique direction with respect to the base image. It is one image. In this case, an 8-direction second-order differential image is obtained, and changes in all angles can be emphasized almost uniformly. Note that nine or more shift images may be acquired for one base image.

In one aspect, the plurality of shifted images are four images that are shifted in two directions in the x direction and two directions in the y direction with respect to the base image. In this case, a quadratic second-order differential image is obtained, and vertical and horizontal changes can be emphasized.

In one aspect, the plurality of shifted images are two images shifted in two directions, the x direction and the y direction, with respect to the base image. In this case, a bi-directional second-order differential image is obtained, and vertical or horizontal changes can be emphasized.

When the contour is simply to be emphasized, in particular, in a case where only a part of the target contour is to be extracted, a difference image (from one base image and one shift image is used using a shift image in only one direction. = Difference integrated image) may be obtained. In actuality, for example, the left and right of the object are emphasized with a plus (white) and a right with a minus (black). In other words, there is also an effect that the sun looks from an oblique direction and looks more three-dimensional. A calculation example of the one-direction shifted image is shown in FIG.

The shift image may be rotated with respect to the base image. For example, for an object that is moving at a constant angular velocity at a constant speed, only the area of the object that is shifted by about the “spatial resolution of the device” (within a predetermined amount) by adjusting the imaging timing. It will be effective. That is, by controlling the timing of imaging, imaging in all areas is possible. At this time, it should be noted that the second-order differentiation is only in the tangential direction, and it can be used for the purpose of observing only a specific pattern. Regarding the adjustment of the imaging timing, for example, using a stepping motor or servo motor, stop the rotation at a certain angle, and then shift to the next angle to capture the image. There is a way to shoot continuously. In the latter case, within the exposure time (detection time) of the detector, it is necessary to rotate the target at such a low speed that the object is considered to be almost stationary, or to shorten the detection time. The term “still” as used herein means that the object moves only within a negligible range with respect to the shift amount defined by the spatial resolution within the detection time. In order to shorten the detection time, the shutter speed can be increased and the exposure time can be shortened, or the actual exposure time can be shortened by irradiating a sufficiently short pulse light with the shutter open. Also good. If it is a method of stopping rotation at a fixed angle with a stepping motor or the like, it is sufficient to take a base image and two shift images on both sides thereof. Rotating back and forth to acquire shift images on both sides (base image → right shift image → left shift image) may be acquired continuously in one direction (right shift image → base image → left shift image). . In addition, when continuously imaging an object that rotates at a constant speed, the base image and the two shifted images on both sides thereof may be acquired continuously while rotating in one direction at a low speed.
[A-7] Other embodiments

As a method for optically acquiring a two-dimensional image, it is roughly divided into:
1. A method of scanning on a two-dimensional surface using a detector (photomultiplier tube, etc.) with one sensor part (0 dimension),
2. A method of scanning in a direction perpendicular to the array (long axis) using a one-dimensional sensor array,
3. There is a method of acquiring an image at a time using a detector (CCD or the like) having a two-dimensional image sensor, and this embodiment is typically applied to Method 3. However, other embodiments of the present invention may also be applied to technique 2. Specifically, the sensor array is shifted by about the spatial resolution in the array (major axis) direction, and then the object is shifted in the vertical direction by about the detection width of the array in one direction. A second-order differentiated two-dimensional image can be acquired. In this connection, reference can be made to Application 3 to an inspection apparatus using a belt conveyor, which will be described later.

The image processing apparatus of the present invention will be described based on an embodiment as an inspection apparatus using a belt conveyor. Such an inspection apparatus can be used for inspecting the shape of a target on a belt conveyor, the presence or absence of a delicate flaw on a surface, such as a liquid crystal panel, a silicon substrate, or an electronic device. For example, if the belt conveyor is sufficiently slow relative to the speed at which the base image and the shift image are acquired, the object is simply carried directly under the image acquisition unit of the inspection device by the belt conveyor, and the object is stationary. Then, the “optical system + detector” of the image acquisition unit can be shifted, and the acquired images can be integrated by difference. More specific description will be given below.

In the application 1 to the inspection apparatus using the belt conveyor, the shift for acquiring the shift image is performed by moving the belt conveyor (the moving direction is X). Application 1 will be described with reference to FIG. The object is placed and conveyed on the belt conveyor, and the movement of the belt conveyor is stopped immediately below the image acquisition unit of the inspection apparatus to make the object stationary and take a two-dimensional image. Next, the object is shifted in one direction by the belt conveyor to the extent of spatial resolution, and is stationary and imaged. This is repeated to obtain a base image and a shift image (one or two images). Alternatively, by capturing images at a high speed to such an extent that the object can be considered to be stationary, the object moving on the belt conveyor is continuously imaged to obtain a base image and a shift image (one or two images). To do. By acquiring the difference image between the base image and the shift image, a two-dimensional image obtained by second-order differentiation only in the left-right direction (X direction) can be acquired.

In the application 2 to the inspection apparatus using the belt conveyor, the shift for acquiring the shift image is performed by the movement of the belt conveyor (X direction) and the shift of the two-dimensional image sensor (Y direction). Application 2 will be described with reference to FIG. Similar to Application 1, the object is placed on the belt conveyor and transported, and the movement of the belt conveyor is stopped immediately below the image acquisition unit of the inspection device, the object is stopped, and a two-dimensional image is captured. Next, the object is shifted in one direction by the belt conveyor to the extent of spatial resolution, and is stationary and imaged. Next, the object is imaged while shifting the two-dimensional image sensor of the image acquisition unit in a direction perpendicular to the traveling direction of the belt conveyor (Y direction) by a slight amount of spatial resolution with respect to the stationary object. The object is shifted in one direction about the spatial resolution by the conveyor, and the base image and the shifted image (four images) are acquired. Alternatively, the object moving on the belt conveyor is continuously imaged by shifting and imaging the two-dimensional image sensor at high speed, and a base image and shift images (four images) are acquired. Thereby, a two-dimensional image second-order differentiated in the XY direction can be acquired.

In the application 3 to the inspection apparatus using the belt conveyor, the shift for acquiring the shift image is performed by shifting the sensor array in the major axis direction (Y direction). Application 3 will be described with reference to FIG. The target portion is imaged while shifting the sensor array in the major axis direction of the array by spatial resolution, and a base image and a shifted image (one or two images) are acquired. Further, the object is shifted in one direction by a belt conveyor in the direction perpendicular to the array in the same direction as the detection width by the array, and the same operation is repeated. Alternatively, by shifting and imaging the sensor array at a high speed, the object moving on the belt conveyor is continuously imaged, and a plurality of base images and shift images are acquired. Thereby, a two-dimensional image obtained by second-order differentiation only in the Y direction can be acquired. In Application 3, the base image and the shift image are narrow-width images corresponding to the sensor array. A difference integrated image of the base narrow-width image obtained by the sensor array and the shifted narrow-width image obtained by shifting one or both sides in the array direction becomes a second-order differential narrow-width image in the shift direction. This operation is repeated by moving the belt conveyor by the detection width of the sensor array, and the second-order differential narrow-width images are joined together to obtain a two-dimensional difference accumulated image (second-order differential image in the array direction). The width of the sensor array is one column or a plurality of columns. In application 3, the object is shifted by the belt conveyor only for joining the narrow images. In particular, it is considered advantageous in inspection of a long object.

Next, a case where the target is moving (for example, a case where a living organism is moving) will be considered. There are two possible approaches for applying the present invention to moving objects. The first approach is a method of acquiring a shift image that is sufficiently fast with respect to the moving speed of the organism. This method shifts the “optical system + detector”, but it is important that the speed of shifting and imaging is sufficiently high relative to the speed of the object, and that the object can be regarded as almost stationary. is there. The second approach is a method of replacing the movement of the object with the shift amount. In this case, the shift of the object is realized by the self-motion of the object, the apparatus does not need to be shifted, the timing of detection is controlled, and the calculation of the acquired image (the concept of direction is eliminated, so the images acquired at different times Is calculated). However, the detection time (exposure time) needs to be short enough to obtain a target still image. By applying this method, it is possible to emphasize and extract the contours of only moving objects at a certain range of speed from a plurality of organisms that move randomly. In this case, due to the time interval of the shift image, the outline of the moving object is emphasized when the interval is short and the object slowly moving is emphasized when the interval is long. A stationary object is completely erased when viewed in the difference accumulated image. In other words, when taking images at a certain time interval and taking a difference image, only the outline of the object that has moved by about the spatial resolution of the device at that time interval is emphasized, faster objects become blurred backgrounds, and slower objects are erased. Will be.

[B] Experimental Example 1
[B-1] Outline of Experiment A spatial second-order differential imaging apparatus using a piezo drive stage was manufactured. FIG. 12 shows an experimental apparatus. This is a device for detecting a change in the optical signal from the sample surface as a two-dimensional image with high sensitivity by combining a CCD camera, an image fiber optical system, and a piezo element driving stage. The image fiber is a non-aligned type, but an aligned type image fiber may be used. A high-accuracy 3-axis piezo element drive stage with a Z axis for focus adjustment and a controller were selected as the drive stage. In addition, a Peltier-cooled CCD camera for astronomical observation with low noise was adopted as the detector. As the light source, a halogen lamp was selected, and red light was selected and irradiated by a band pass filter. By slightly changing the position of the sample directly under the image fiber with the piezo element drive stage and taking the difference of the acquired images, the temporal and spatial fluctuations of the light source, fiber optical system, and detector are almost cancelled. Only the position dependence of the optical signal is detected with high sensitivity.

Only the sample position dependency of the signal intensity is extracted by shifting the entire measurement system including the optical system and the detector by a slight amount relative to the sample. The difference integrated image generated by this operation becomes a spatial second-order differential image. The sample is on a piezo drive stage and can be moved in a small amount in the XY plane by the piezo drive stage. Only the sample is shifted a fixed amount 4 times (or 8 times) with respect to the optical system and detector, and each image data (4 shift images in the case of 4 shifts, 8 images in the case of 8 shifts) Shift image). 13 shows a base image _{C 0} and the eight vertical and lateral oblique 8-direction shift image _{C 1 to 8.} Assuming that the base image C ₀ is in the XY plane, the shift images C ₁ and C ₂ are images shifted in two directions in the X direction, the shift images C ₃ and C ₄ are images shifted in two directions in the Y direction, and the shift image C It can be said that _{5 to 8} are images shifted in four directions in an oblique direction. Each hexagon which comprises the honey-cam shape in FIG. 13 shows a fiber strand. The shift amount in FIG. 13 is an amount corresponding to the diameter of the fiber strand of the image fiber. The signal intensity of the base image is 4 times (or 8 times for 8 shift images) in the case of 4 shift images, and 4 (or 8) shift images obtained by shifting are used. By subtracting from the signal intensity integration, a spatial second derivative image of the base image is obtained.

[B-2] Experimental result The shift amount dependency between the difference accumulated image and the sharpened image is shown in FIG. 14 (shift amount: 0 μm, 0.25 μm, 0.5 μm), FIG. 15 (shift amount: 1 μm, 2 μm, 3 μm). FIG. 16 (shift amount: 4 μm, 5 μm). 14 to 16, the upper diagram (−2 ^nd derivative) is a difference integrated image, and the lower diagram (0 ^th −2 ^nd derivative) is a sharpened image. Edge information can be obtained even with a shift amount of about 1/10 (0.25 μm) of the spatial resolution. Around 2 or 3 μm seems to be optimal. If the shift amount is more than 4 μm or 5 μm, the contour becomes thick and the image in the central portion can be discriminated, although the image is somewhat unnatural. The optimum shift amount is considered to be about 2-3 μm (the diameter of the fiber strand) with a resolution.

As shown in the observation result of the graphene hole bar in FIG. 17, single-layer graphene having the same width as the fiber diameter could be identified. The single-layer graphene cannot be determined by numerical calculation using a Laplacian filter.

As shown in the observation result of single-layer graphene / graphite in FIG. 18, single-layer graphene and a complicated stacking state could be clearly observed. Markers could also be identified. In the numerical calculation using the Laplacian filter, the marker cannot be identified.

As shown in the observation result of single-layer / double-layer graphene in FIG. 19, the laminated state of single-layer and double-layer graphene could be clearly observed. In the numerical calculation using the Laplacian filter, the lamination state cannot be determined.

[B-3] Summary • A spatial second-order differential imaging apparatus using a piezo drive stage was manufactured.
-Observation by reflected light was performed using single layer and multilayer graphene samples.
・ As a result, it was confirmed that the resolution was clearly improved by this method.
• The optimum shift amount of the drive stage was about the diameter of the fiber strand of the imaging fiber that determines the spatial resolution in this device.
-It was confirmed that this improvement in resolution was not realized by second-order differential image processing (Laplacian filter) by numerical calculation.

[C] Experimental example 2
[C-1] Outline of Experiment A spatial second-order differential imaging technique using position shift was applied to a general optical microscope. Olympus BX60 was used as the optical microscope. Because of the screw-type manual stage, a strict shift is not possible. The shift amount this time is approximately 1 ± 0.5 μm.

When estimating the resolution of the microscope used this time, the R component (red) of the RGB output of the CCD was used, and the numerical aperture of the objective lens was 0.4.
Therefore, it is considered to be about 1 μm.

[C-2] Experimental Results FIGS. 20 and 21 show the observation of single layer / multilayer graphene.
-Single-layer graphene could be observed more clearly than the base image.
-For multilayer graphene, the lamination state became clearer than the base image.
-Markers in multilayer graphene were clearly observed. In the conventional method, the numerical operation using the Laplacian filter is finally performed until it can be seen.
・ A clearer image was obtained than numerical computation using the Laplacian filter.

22 and 23 show the observation of single layer / multilayer graphene.
-Single-layer / double-layer graphene could be observed more clearly than the base image.
・ As for multilayer graphene, a pattern that seems to be due to lamination was visible. In the conventional method, even if a numerical calculation using a Laplacian filter is performed, it is hardly visible.
・ A clearer image was obtained than numerical computation using the Laplacian filter.

[C-3] Summary • A spatial second-order differential imaging technique using position shift was applied to an optical microscope.
・ Due to the use of a screw-type manual stage, an accurate shift was not achieved (approximately 1 ± 0.5μm), but a clear resolution improvement was confirmed by this method.
-Confirmed that information that cannot be obtained by numerical calculation using Laplacian filter can be obtained.
・ It is thought that the resolution will be improved if it is shifted accurately with the spatial resolution of the device.

[D] Experimental example 3
A comparative experiment was performed on the weight k of the base image C ₀ when the sharpened image C was acquired (C = kC ₀ −C _{difference accumulation} ). Experimental results are shown in FIGS.
1 ≦ k: The contour is emphasized up to about k = 20, and the effect of position shift is not so much above that.
0 ≦ k <1: There was no significant difference from the difference integrated image.
k <0: The pixel intensity changes relatively, and the uneven shape also changes. It seems that it is not very effective.
Although not limiting the present invention, 1 ≦ k ≦ 20 is considered appropriate.

Claims (9)

An image processing method using an image acquisition device for acquiring a target image,
Acquiring a target image as a base image;
Obtaining N shift images (N ≧ 1) obtained by shifting the object by a predetermined amount in the imaging plane with respect to the base image;
Obtaining N difference images (N ≧ 1) from the difference between pixels of the same coordinates of the base image and each shift image;
Adding the N difference images to obtain a target difference integrated image;
Tona is,
The difference integration image is obtained by subtracting the base image from each shift image and integrating,
Furthermore, the image processing method which comprises the step which subtracts the said difference integration image from a base image, and acquires the sharpening image of object . The image processing method according to claim 1 , wherein the predetermined amount is an amount in a range of 1/10 to 2 times a spatial resolution of the image acquisition device. The image processing method according to claim 2 , wherein the predetermined amount is an amount in a range of 1/5 to 1 times a spatial resolution of the image acquisition device. The shift image is subject or / and is obtained by moving the measuring system of the image acquisition apparatus, an image processing method according to any one of claims 1-3. The N shifted images are
8 images shifted in 2 directions in the x direction, 2 directions in the y direction, and 4 directions in the diagonal direction with respect to the base image, or
4 images shifted in 2 directions in the x direction and 2 directions in the y direction with respect to the base image, or
Two images shifted in two directions in the x direction or two directions with respect to the base image, or
It is a single image, which is one direction shifted in one direction or the y-direction in the x-direction relative to the base image, the image processing method according to any one of claims 1-4. The shift image is an image obtained by rotating the base image, the image processing method according to any one of claims 1-3. An image processing apparatus using an image acquisition apparatus for acquiring a target image,
Means for acquiring a target image as a base image;
Means for acquiring N shift images (N ≧ 1) in which the object is shifted by a predetermined amount within the imaging plane with respect to the base image;
An image processing unit,
The image processing unit acquires N difference images (N ≧ 1) from a difference between pixels having the same coordinates in the base image and each shift image, and adds the N difference images to add a difference of a target. Get an image ,
The image processing unit obtains the difference accumulated image by subtracting and integrating a base image from each shift image, and obtains a target sharpened image by subtracting the difference accumulated image from the base image.
Image processing device. The image processing apparatus according to claim 7 , wherein the predetermined amount is an amount in a range of 1/10 to 2 times a spatial resolution of the image acquisition device. The image processing apparatus according to claim 8 , wherein the predetermined amount is an amount in a range of 1/5 to 1 times a spatial resolution of the image acquisition device.