JPS63246716A - Method for taking-in of endoscope image - Google Patents

Abstract

PURPOSE:To permit easy taking-in of images together with factors to associate the relations between the plural images by taking in the 2nd image superposely on part of the 1st image. CONSTITUTION:The endoscope image which is picked up by a TV camera 12 and is digitized is recorded in a 1st frame memory 14 by the instruction from a recording instruction means 26. The value of a counter 23 of this time is recorded in a 1st latch 24 simultaneously therewith. Then, an operator slightly rotates an angle knob 20 to put the curved part 5 of the endoscope to a 2nd curved state. The digitized endoscope image in the position of the 2nd curved state is recorded in a 2nd frame memory 15 by using the means 26 in this state. The value of the counter 23 is stored in a 2nd latch 25 simultaneously at this time. The height between the two points of a specimen 4 and the information on the ruggedness of the specimen 4 are determined in accordance with the above-mentioned information and are displayed in an image processor 16. The easy taking-in of the plural images together with the information to associate the relations between the images is thereby permitted.

Description

[Detailed description of the invention] [Industrial application field] The present invention relates to an endoscopic image capture method.

[Conventional technology]

Conventionally, known endoscopes include fiberscopes that transmit images through image fibers, and electronic scopes that have a solid-state image pickup device at their tips, and have been mainly used for image observation. On the other hand, if image recording is required, this can be done by recording photographs with a camera connected to the eyepiece of the fiberscope, or by recording signals from the electronic scope in an image memory.

[Problem that the invention seeks to solve]

Conventional photographic recording and image recording record a single image at the time of recording, and also record a plurality of images one after the other, but there is no relationship between the images. Therefore,
Even when a recorded image is processed, various processes are only performed using information within a single image.

The present invention has been made in view of the above-mentioned problems, and it is an object of the present invention to provide an endoscopic image capture method that can easily capture a plurality of images together with factors that relate their mutual relationships.

[Means and effects for solving the problems] The present invention has the following features:
Capturing a first image at a first position via an endoscope; capturing a second image at the second position while detecting the relative position of the second position with respect to the first position; The second image is characterized in that a portion of the image overlaps with the second image.

〔Example〕

Hereinafter, the present invention will be described in detail with reference to the drawings. 1st
The figure is a configuration diagram showing a first embodiment of the present invention. An objective lens 1 is provided at the distal end of the curved portion 5 of the endoscope. This objective lens 1 has a function of forming an image of the subject on the end face of the image guide fiber 8. Further, a concave lens 7 is provided at the tip of the curved portion 5.

This concave lens 7 has a function of emitting light that has passed through a light guide 6 made of a glass fiber or the like for guiding illumination light from the outside with good distribution. The other end of the image guide fiber 8 is guided to an eyepiece 10 formed in the operating section of the endoscope, and is imaged by a TV camera 12 via an eyepiece (not shown) and an imaging lens 11. An image formed on the endoscope end side of the guide fiber 8 is captured. The output of the TV camera 12 is converted into a digital signal by an A/D (analog/digital) converter 13, and is stored in the first frame memory 1.
4 or the second frame memory 15. 1st and 2nd frame memo! The output of 714.degree. 15 is manually input to an image processing device 16, the details of which will be described later. On the other hand, an angle wire 17 for bending the curved portion 5 of the endoscope is wound around a rotating drum 18 disposed within the operating section of the endoscope. The rotating drum 18 includes a pattern disc 19 disposed within the operating section.
and is fixed coaxially with the angle knob 20 which is exposed and disposed in the operation section. A black and white pattern at equal intervals is formed on the circumferential end surface of the pattern disk 19.
It can be read by a sensor 21 composed of a reflective photosensor or the like. The output of the sensor 21 is input to an amplifier 22, and the output of the amplifier 22 is input to a counter 23. The output of the counter 23 is
The first latch 24 and the second latch 25 are manually operated, and their outputs are input to the image processing device 16. The output of the recording instruction means 26, which is composed of operation switches, timing circuits, etc., is inputted to the first and second frame memories 14,15, and the first and second latches 24,25.

Next, the operation of the above embodiment will be explained. As shown in FIG. 2, when the endoscope curved section 5 is in the first curved state (in this embodiment, the uncurved state shown by the solid line), the recording instruction means 2
According to the instructions from 6, the image is captured by the TV camera 12, and the A/
The endoscopic image digitized by the D converter 13 is the first
It is recorded in frame memo 1J14. At the same time, the value of the counter 23 at this time is recorded in the first latch 24. Next, turn the angle knob 20 slightly to adjust the curved portion 5 of the endoscope.
is brought into the second curved state shown by the broken line in FIG. At this time, as the angle knob 20 rotates, the pattern disk 19 also rotates, and the sensor 21 detects a change in the black and white pattern.

That is, the pattern disk 19 constitutes a rotary encoder.
At the same time as the amount of rotation, information on the direction of rotation is also transmitted to the counter 23 via the amplifier 22. In this second curved state, a digitized endoscopic image at the position of the second curved state is recorded in the second frame memory 15 using the recording instruction means 26. At this time, the value of the counter 23 is simultaneously stored in the second latch 25.

In this way, the subject 4 in contact with the shaded area in FIG.
This means that two types of images were obtained with the parallaxes shown in FIG. 2 A'-B'. Also, the positional relationship between these two types of images is stored.

The images stored in the first and second frame memories 14 , 15 are supplied to an image processing device 16 . Further, the values stored in the first and second latches 24 and 25 are supplied to the image processing device 16 and converted into angle information θ. Based on this information, the image processing device 16 calculates and displays the height between two points on the subject 4 and unevenness information on the subject 4, as will be described later.

Although this embodiment has been described with respect to an endoscope having an image guide fiber, it goes without saying that the present invention may also be applied to an electronic scope having a solid-state image pickup device at the distal end of the endoscope. In this case, after signal processing and A/D conversion of the image signal obtained from the solid-state image sensor provided at the tip of the endoscope, the first and second frame memo 1J1
4.15 may be configured to store images at two positions.

An embodiment in this case is shown in the third embodiment (FIG. 4) described later.

Next, a second embodiment will be explained using FIG. 3.

As shown in FIG. 3(A), a camera 27 is attached to the endoscope eyepiece 10 so that the image of the other end of the image guide fiber 3, that is, the endoscopic image, is formed on a film 28. It is located. The camera 27 has a data recording means 2 consisting of an LED and the like for recording data on a film 28.
9 is provided, and the data ear language control means 29 is connected to the data ear language control means 30. On the other hand, a slit disk 60 is used instead of the pattern disk 19 in the first embodiment.
is directly connected to the angle knob 20 and the rotating drum 18, and the LED driver 31 emits light.
The light from D32 is arranged so as to cross the slit disk 60 and reach the light receiving element 33 composed of a phototransistor or the like. The output of the light-receiving element 33 is inputted to the counter 23 via the amplifier 22, and the output of the counter 23 is connected to the data-speech control means 30. Further, a release instruction means 34 composed of a release switch or the like is also connected to the data ear-speech control means 30.

The film 28 on which the endoscopic image was photographed by the camera 27 is developed, and as shown in FIG. 3(B), the film reading means 3
It is read in 5. That is, the first slide 37 is photographed together with the endoscope image with the amount of curvature of the endoscope curving portion 5 in the first state as the first curvature amount information 36, and the amount of curvature of the endoscope curving portion 5 in the second state is The amount of curvature in this state is used as second curvature amount information 38, and the second slide 39 photographed together with the endoscopic image is read separately by a film reading means 35 constituted by a drum scanner or the like. The output of the film reading means 35 is sent via the A/D converter 61 to the data discriminating means 4 for discriminating between the endoscopic image and the curvature amount data.
Input to 0. The image data 41 and curvature amount data 42 that are output from the data discrimination means 40 are processed by the image processing device 16.
is input to. The output of the image processing device 16 is sent via a D/A converter 43 to a monitor 44 such as a TV monitor.
is input to.

Next, the operation of the second embodiment will be explained. When the endoscope curved section 5 performs a release operation using the release instruction means 34 in the first curved state (the state indicated by the solid line in FIG. 2),
The data reading control means 30 drives the data reading means 29 to transfer the value of the counter 23 at that time onto the film 28. In this case, the value of the counter 23 may be recorded as a number as it is, or may be encoded and recorded like a bar code.

Next, wind up one frame of film 28, and turn the angle knob 20.
is slightly rotated to bring the endoscope curved portion 5 into the second curved state. At this time, the slit disk 60 also rotates as the angle knob 20 rotates, and the light receiving element 33 detects that the light from the LED 32 is interrupted by the slit disk 60.

By arranging two sets of LEDs 32 and light receiving elements 33 at positions where the slit patterns of the slit disk 60 are shifted by 90°, the rotation direction can also be determined. Due to this bending operation, the value of the counter 23 also changes as the slit disk 60 rotates. When a release operation is performed using the angle release instruction means in this state, the value of the counter 23 is recorded on the film 28 together with the endoscopic image by the data release control means 30 and the data reference means 29. Through the above operations, the first slide 37 on which the amount of curvature and the endoscopic image in the first curvature state are photographed, and the second slide 39 on which the amount of curvature and the endoscopic image in the second curvature state are photographed. is obtained. The first slide 37 and the second slide 39 are applied to the film reading means 35, and the output signal thereof is converted into a digital signal by an A/D converter 61. The data discriminator 40 discriminates between the endoscopic image signal and the curvature amount signal from among the digital signals, and separates the image data 41 and the curvature amount data 42 and inputs them to the image processing device 16.

Based on the input data, the image processing device 16 calculates the distance to the subject 4 and the unevenness of the subject 4, as will be described in detail later, and displays a stereoscopic composite image on the monitor 44 via the D/A converter 43. Display.

Furthermore, a third embodiment will be explained using FIG. 4.

This third embodiment includes the image guide fiber 8, the endoscope eyepiece RIO1 imaging lens 11, and the TV shown in the first embodiment.
The imaging system consisting of the left camera 2 is replaced with an electronic scope having a solid-state imaging device 45 disposed at the tip of the endoscope. Objective lens l (not shown) is for object 4 (not shown)
is arranged so as to form an image on the solid-state image sensor 45, and its output is outputted via a video amplifier 50 and an A/D converter 51.

In addition, a short pattern 46 is attached to the shaft of the angle knob 20.
A short pattern disc 47 having a pattern 46 on its surface is provided, a contact brush 48 for detecting the pattern 46 is provided, and the contact brush 48 is connected to a drum position detection means 49. The short patterns 46 are provided at intervals such that the distance that the tip of the endoscope curved portion 5 moves at equal intervals when the angle knob 20 is rotated. The contact brush 48 is provided with three brushes in order to know the direction of rotation of the short pattern disk 47. These three brushes are positioned so as to be short-circuited with respect to the short pattern 46, and the direction of rotation can be determined by the order of the timing at which each set of two of the three brushes short-circuits. It has become. From the drum position detection means 49, the first frame memory 1
A recording instruction signal 52 is supplied to the frame memory 4 and the second frame memory 15.

Next, the operation of this third embodiment will be explained. As the angle knob 20 rotates, each time the contact brush 48 is short-circuited by the adjacent short pattern 46, the drum position detection means 49 captures an image in the first frame memory 14 or the second frame memory 15 by the solid-state image pickup device 45. A recording instruction signal 52 is issued to record the endoscopic image. Either record it in the first frame memory 14 or record it in the second frame memory 15.
You can toggle whether or not to record. As a result, the first frame memory 14 and the second frame memory 15 can always record endoscopic images with a constant parallax. Both frame memories 14 and 15 are input to the image processing device 16, and the details will be described later, but the image processing device 16 calculates the distance to the subject from these images and the shod pattern 460 interval, that is, from a constant parallax value. Calculate the state of unevenness,
It is displayed on the monitor 44 via the D/A converter 43.

Furthermore, a fourth embodiment will be explained using FIG. 5.

This fourth embodiment corresponds to an electronic scope in which a solid-state image sensor 45 is disposed at the tip of an endoscope, and the output of the solid-state image sensor 45 is transmitted to a video amplifier 50, an A/D converter 51, and a frame memory 53. The signal is input directly from the A/D converter 51 to the image processing device 16 via the A/D converter 51.

On the other hand, in order to obtain the amount of curvature of the endoscope curved portion 5, that is, the value of parallax, a black and white pattern is provided on the angle wire 17, and the amount of movement of the wire 17 is detected by a sensor 21 composed of a reflective photosensor or the like. . The output of the sensor 21 is connected to an amplifier 22 and a counter 23, and the output of the counter 23 is connected to a latch 54 and an image processing device 16.
The output of is also connected to the image processing device 16. Further, a recording instruction means 55 is provided which is operated by a signal from an operation switch (not shown) and outputs an operation signal to the frame memory 53 and the latch 54.

Next, the operation of this fourth embodiment will be explained. This fourth embodiment uses two frame memories as in the first embodiment when the processing speed of the image processing device 16 is high or when the image processing device 16 has internal frame memories and latches. This shows that it can be implemented without using a latch. That is, when the endoscope bending section 5 operates the operation switch in the first state, the value of the counter 23 in the first state is recorded in the latch 54 by a signal from the recording instruction means 55, and the solid-state image sensor 45 The endoscope image captured in the above is recorded in the frame memory 53. On the other hand, the endoscopic image when the endoscope bending section 5 is in the second state and the value of the counter 23 are directly taken into the image processing device, and the endoscopic image and the counter value when the endoscope bending section 5 is in the first state are processed in real time. The distance to the object and the state of unevenness are calculated and displayed on the monitor 44 via the D/A converter 43.

Next, the principle of measuring the absolute values of the height and size of an object in the image processing device 16, which will be described later in the embodiment, will be explained first.

As shown in Figure 6, if the bending angle of the endoscope tip when capturing two images is θ and the radius of curvature is r, the distance l between the objective lenses before and after bending is θ. When is small, p=r
・Approximate by θ. Next, FIG. 7 shows the geometric correspondence between the two images and the target object. In addition, in FIG. 7, in order to simplify the explanation, the optical axis direction of the lens is defined as 0.0, the center of the lens before and after the movement. ,
This will be explained assuming that it is a direction perpendicular to the straight line connecting 0 degrees. The position of the center of the lens in the two states is 02,0
□, the focal length of the pHP2% lens of the image obtained at that time is f. Arbitrary points A and B on the object are mapped at positions al+bl on image P1, and at positions a2+b2 on image P2. Here, if the two images are superimposed with their centers aligned, the point al+bl corresponds to al'+b,' on image P2. At this time a,
The distance between / and 82 is 'a', and the distance between bl' and b2 is d
, and so on. Also, 0. , 02 and is parallel to the target object.

The distance between Po and point A is , and the distance between Po and point B is h8
shall be. Then, the following relational expression holds true between hA and d due to the similarity between triangle A010□ and triangle 0□a,'a2.

Therefore, ha can be determined using the following equation.

Similarly, ha+ can be found using the formula 0.

In this way, by determining the distance d between corresponding points in the two images PII P2, the absolute size h of the point in the height direction can be determined.

Next, we will show how to find the absolute size of the distance between arbitrary points. The center of image P1 is C3, and 0 on P1
Let the distance between 1 and a be ea, and the distance between cI and bl be e. Further, if the distances between a straight line I drawn perpendicularly from 01 onto the object and points A and B are WA and W, then the following holds true between WA and e6 due to the proportional relationship. Therefore,
Substituting the formula ■ into the formula ■, L can be found using the formula 0.

Similarly, Wa can be found using the formula 0.

b Therefore, the object is P. Point A when projected onto a plane parallel to
The distance 1IIAB between and point B can be determined using the formula (2).

Therefore, the absolute magnitude of the distance between arbitrary points on the image P can be measured.

Next, a method for finding the distance d between corresponding points in the two images P1 and P2 will be explained. Basically, it is determined by examining the correlation in a small area in two images. The small area in the two images is defined by the function f (tr),
g(tr), and g(tr) is shifted by (D compared to f(tr). In other words, g(tr)=f
Assume that (+r-10). However, tr is a coordinate representing two dimensions.

The correlation between f (tr) and g (tr) is defined by the following equation.

ψ(s) = L f(sr) −g ” (tr−s
) −d tr −−１■(Hereinafter, ψ(s) = f(
(abbreviated as g(tr)), where A represents the area of the small region.

The Fourier transform of Equation 0 is shown in Equation (2).

Φ(u+) = F(u+) ・G I(Iu) −
---■ However, F(u) is a function obtained by Fourier transform of f(''), and G(1) is a function obtained by Fourier transform of g(tr).

Here, using the condition of g(tr)・f(+r−[)), the formula ■ becomes Φ(lu) = F(Iu) 4 (iu) −e−”
''''4'4 ---.When the formula is inversely Fourier transformed, it becomes the formula 0.

ψ(s) = Rye(1) *δ(t-10>
-------111) However, Rrr(t) is f(r
) is the autocorrelation function of Lrfi) = f(tr) *f(tr) ---
--Represented by @. Also, e-J-21'11'II
We used the fact that the inverse Fourier transform of is δ-]).

Equation 0 indicates that the correlation function ψ($) has a peak at $=0. Therefore, the correlation function ψ(S
), and by checking the position of the peak, g
It is possible to determine how much (r) is shifted with respect to f('').Using this fact, find the corresponding small area from the two images P,, P, and calculate the corresponding The distance d between points can be found.

Next, an actual configuration example will be explained.

FIG. 8 shows a block diagram of the entire processing system related to the above processing in the image processing device 16. Two pieces of color image data 90 sent from the endoscope are recorded in an image memory 91. Image data in the image memory 91 is input to a color-monochrome converter 92 and converted into monochrome image data suitable for measurement. The output from the color-monochrome converter 92 is corrected for image distortion by a distortion corrector 93 while referring to the curvature angle θ. The two images whose distortion has been corrected are input to a correlation calculator 94 to detect the distance between corresponding points, that is, the amount of shift, and this is stored in a shift amount memory 95. The image display device 96 displays the image recorded in the image memory 91, the unevenness information of the image calculated from the shift amount and the value of θ recorded in the shift amount memory 95, and also the unevenness information of an arbitrary part in the image. Displays absolute values of size and height.

(1) Image input Two color image data captured by bending the tip of the endoscope are input and stored in the image memory 91. Further, the value of the curve angle θ is input to the distortion corrector 93 and the image display device 96.

(2) Correction of image distortion The input image is distorted by taking it with a wide-angle lens or by tilting the tip of the endoscope, so this is corrected. As shown in Figure 9, the method is to first take a picture of a square square on a plane under the same conditions before taking an image inside the body, and then determine the correction value for each pixel so that the image becomes a square. put. Then, correction is performed on the image that is actually input.

A block diagram of the image distortion corrector 94 is shown in FIG.

The value of θ is input to the memory 104 which stores the distortion correction value at θ which can be set, and the image data is recorded in the image memory 105 while correcting the distortion by giving an appropriate address to the human image data. do.

Next, an interpolation calculator 106 performs an interpolation calculation on the image recorded in the image memory 105 to prevent high spatial frequency components from deteriorating due to distortion correction. This interpolation function is a 5-pline function as shown in Figure 11, which has a form close to a 5-inc function, f(X)=lXl'-
21Xl”+1 1XI<1f(X)=-IX13+
5 1XI2-8 1Xl+41≦lXI<2 f(X)=Qlxl≧2, etc. are used.

(3) Correlation Calculation In the correlation calculation, as shown in FIG. 12, a detection point P and a calculation target area centered around the detection point KP are set in the two left and right images LP and RP. Initially, the calculation target area is set to be a somewhat large area, for example BB, in which the correlation peak is not very sharp, but in order to get a rough idea, then the calculation is performed on a small area SH after narrowing down the target area.
Determine the corresponding area with high accuracy.

(i) First example of a correlation calculator using an electric circuit FIG. 13 shows the configuration of a first example of a correlation calculator using an electric circuit. The address generator 108 specifies the area to be calculated in the image memory 107, and the sum-of-products calculator 1
Correlation calculation is performed using 09. The result is determined by the judge 110
It is judged by. First, when performing a correlation calculation on a large area to be calculated, a correlation value exceeding a certain value and the address at that time are sent to the controller 112 side. Then, the controller 112 judges the result and sends a command to the address generator 108 to generate an address for the targeted area. Correlation calculations are then performed in the same manner for the small set area, and this time only the maximum value is determined by the determiner 110, and the address at that time is sent to the shift amount memory 111.

Although this configuration takes time to calculate, it can be configured with simple parts.

Second Example FIG. 14 shows the configuration of a second example of a correlation calculator using an electric circuit. This is because the FFT is used instead of the product-sum calculator 109 to perform the correlation calculation in the configuration of the first example.
113a, 113b113b and multiplier 114 and inverse F
This uses an FT device 115. In addition, FFT113
b outputs a complex conjugate value after Fourier transformation.

By using this circuit, calculation speed can be increased.

Third Example FIG. 15 shows the configuration of a third example of a correlation calculator using an electric circuit. The left and right image signals f and g recorded in the image memory 107 are respectively averaged by an average calculator 120a,
120b to input the average density value f in a predetermined area.
, g, and then their average value f.

Input i to the multiplier 121 to calculate 7.King. Also,
Both left and right image signals f and g are input to a multiplier 122, and f
After calculating -g, this is input to the average calculator 123 to calculate the average value f·g of f−g in one predetermined area. The output r1 of the average calculator 123 is input to the subtracter 124 together with the output cuff layer of the multiplier 121, where the cross-correlation f-g-f-g is calculated. Furthermore, the left and right image signals f, g from the image memory 107 and the output signals f, g of the average calculators 120a, 120b are input to standard deviation calculators 125a, 125b, respectively, to calculate the standard deviation of both images in a predetermined area. σ,..."6 units, 7 units, σ
9.4? −(i)2, and input these to the multiplier 126 to calculate σ,·σ. The outputs of the divider 126 and the subtracter 124 are manually input to the multiplier 127 to calculate f-g-f-g σf'' σ9, which is input to the determiner 110.The following is the same as the above example. , a determiner 110, a controller 112,
The address generator 108 determines the corresponding area and sends the address of the correlation peak to the memory 111 for recording.

In this example, when performing a correlation calculation using a product-sum calculator, the average value is subtracted for each of the left and right images,
I try to normalize using the standard deviation. In other words, the circuit is constructed based on σf ・σ9, which is expressed as the formula, and the bias and gain between the left and right images, which cause poor accuracy when calculating the correlation between the left and right images, are We can cancel out the differences. Therefore, not only can it be constructed with simple parts, but also correlation peaks can be detected with high accuracy.

Fourth Example FIG. 16 shows the configuration of a fourth example of a correlation calculator using an electric circuit. In this example, both the left and right image signals recorded in the image memory 107 are manually subjected to predetermined spatial frequency filtering in the spatial frequency filtering devices 130a and 130b, respectively, and then the product-sum calculator 109
The other configuration is the same as that shown in FIG. 13. Note that the spatial frequency filtering device 1
30a, 130b can be configured as shown in FIG. 17, FIG. 18, or FIG. 19, for example.

In FIG. 17, an image signal from an image memory 107 and an address signal from an address generator 108 are manually input to a look-up table memory (LOT) 131 to output a table conversion value according to the pixel value and address. , this value is added in the adder 132 to the value recorded in the memo 1J133 and recorded in the memory 133, thereby updating the value.

With this configuration, it is possible to perform a sum-of-products calculation for a predetermined area of an image. In addition, memo IJ1
The selector 134 selects whether to return the value recorded in the adder 33 to the adder 132 or to output it to the product-sum calculator 109 in accordance with the address signal from the address generator 108.

In addition, in FIG. 18, instead of using the LOT 131 in FIG.
5 and a multiplier 136 to multiply the pixel signal by a coefficient corresponding to the value of the address and output the result to the adder 132.The other configuration is the same as that in FIG.

FIG. 19 shows a system configured to perform sum-of-products operations on a 3×3 pixel local area at high speed, and image signals are inputted in time series according to the order of raster scanning. Among these image signals, one line delay device 140a, 140
The pixel signal that has passed through both of the 1-line delay devices 140a and 140b is sent to the converter 141C, the pixel signal that has passed only through the 1-line delay device 140a is sent to the converter 141b, and the pixel signal that has passed through both of the 1-line delay devices 140a and 140b is sent to the converter 141C. are manually input to the converter 141a at the same time. The conversion devices 141a to 141C are L in FIG.
tlT 131 or memo IJ13 in Figure 18
5 and multiplier 136, and outputs a value obtained by multiplying a pixel signal by a conversion coefficient corresponding to an address as a conversion value. The outputs from these conversion devices 141a to 141C are input to an adder 142 and added, and the added output is input to a 1-pixel delay device 143a, an adder 144a, and an adder 144b. The signal input to the one-pixel delay device 143a is manually inputted to the adder 144a after delaying the timing by one pixel, and is then input to the adder 144a after one pixel.
42 and one pixel delay device 143b.
Enter. In the one-pixel delay device 143b, the output of the adder 144a is delayed by one pixel and inputted to the adder 144b, where the output of the adder 142 is added and inputted to the product-sum calculator 109 shown in FIG. In this way, the product-sum calculation for 3×3 local regions can be performed at high speed using the pipeline method. Note that in FIG. 19, the product-sum operation is performed for a 3×3 local area, but it is also possible to perform a product-sum operation of an even larger size by increasing the number of each element using the same method.

As described above, in this fourth example, spatial frequency filtering is performed on each image before performing the correlation calculation using the product-sum calculation. here,
Image correlation can be determined more accurately by extracting regions of specific spatial frequency components according to their properties. Therefore, in this fourth example, spatial frequency filtering is realized by performing a sum-of-products operation for a local region centered on each pixel for each pixel in the image, and then a correlation operation is performed. Regarding this filtering, if the amount of movement of the endoscope tip is small and the correlation between the left and right images is very large, the accuracy of the correlation can be further improved by extracting the high spatial frequency components of the images. . An example of this bypass filtering is 3X3 as shown below.
By processing a pixel region using a Laplacian filter, it is possible to cut low frequency components of an image and extract edge portions.

However, * indicates a combination.

In addition, if the correlation between the left and right images is degraded due to shape distortion due to geometric conditions or lighting conditions, the correlation can be improved by extracting low spatial frequency components using low-pass filtering. Can be done. As an example of this low-pass filtering, processing can be performed using an averaging filter as shown below.

In addition to the above, it is possible to perform product-sum operations using various coefficients, and by increasing the size of the local area, it is also possible to provide a bandpass filter for a specific spatial frequency filter link. Appropriate filtering can be performed depending on the image.

In this fourth example as well, it is possible to perform highly accurate (correlation calculation) with a relatively simple component configuration.

Fifth Example FIG. 20 shows the configuration of a fifth example of a correlation calculator using an electric circuit. In this example, in the circuit configuration shown in FIG. 14, the outputs of FFTs 113a and 113b are subjected to predetermined filtering by spatial frequency filtering devices 145a and 145b, respectively, and then input to a multiplier 114 and processed in the same manner. This is what I did. Spatial frequency filtering devices 145a, 145 in this case
The configurations of two examples of b are shown in FIGS. 21 and 22.

Figure 21 shows the lookup table memo 'J (L[IT
) 146, the frequency dispersion and Fourier transform value F(u
) and transform F(u) based on the conversion table selected by its frequency U, resulting in P' (
u) is output.

Moreover, FIG. 22 uses the LIT 147 and the multiplier 148, and the value of the frequency U from the FFT 113a is
A coefficient k corresponding to the frequency dispersion is input to the tlT 147 and outputted, and this coefficient is multiplied by the Fourier transform value F(+u) from the FFT 113a in a multiplier 148 to obtain F' (a, l). It is designed to output .

Although the configuration of one of the spatial frequency filtering devices 145a is shown in the right side of FIG. 21 and FIG. 22, the other spatial frequency filtering device 145b can also be configured in the same manner.

In this fifth example, as in the fourth example described above, correlation calculation is performed after applying appropriate spatial frequency filtering to the image, but in this fifth example, the image is Fourier transformed and then filtered in the spatial frequency domain. Since this is done, arbitrary filtering is possible.

In other words, there is no restriction due to the size of the local area as in the fourth example, and optimal filtering can be performed according to the image. Therefore, more accurate correlation calculations can be executed at high speed.

Sixth Example FIG. 23 shows the configuration of a sixth example of a correlation calculator using an electric circuit. This example is FFT 113a.

The output of 113b is manually input to the multiplier 114, and at the same time inputted to the absolute value calculators 149a and 149b, respectively, and the absolute value of the Fourier-transformed value in these absolute value calculators 149a and 149b (IFl・,ρ?−F” ), these outputs are multiplied by a multiplier 150, and a division is performed in a divider 151 by the output of this multiplier 150 and the output of the multiplier 114, and the output is inverted F.
It differs from the second example (Fig. 14) described above in that it is input to the FT 115, and the other configuration is the same as the second example.
Similar to the example.

In this sixth example, when performing the correlation calculation via Fourier transform, the images f and g are transformed into two values F,
The correlation calculation is performed after dividing G'' by the absolute value of each. That is, the equation is expressed as follows.

However, τ-1 represents an inverse Fourier transform.

If such a method is used, only the phase information determines the correlation without depending on the Fourier pesctor, and therefore the correlation can be detected with high accuracy. Therefore, when highly correlated stereo images are obtained, such as when the viewing angle of the endoscope objective lens is narrow and the swing angle of the tip is small when capturing left and right images, it is possible to obtain stereo images with high accuracy and high speed. , and the correlation operation can be performed in a relatively simple way.

(ii) First example of a correlation calculator using an optical device FIG. 24 shows a block diagram of a first example of a correlation calculator using an optical system. The output light from the laser 156 is expanded to an appropriate diameter by a beam expander 157, becomes a reference light for the hologram 162, and at the same time is sent to a half mirror 158.
About half the intensity of the light is reflected by the mirror 159, and this reflected light is further totally reflected by the mirror 159 to illuminate the image film 160.

The image film 160 is obtained by printing either the left or right image data stored in the image memory 107 when explaining the distortion corrector 93 onto a positive film, and as shown in FIG. 25, only the calculation target area 08 is printed. It is installed by covering it with a mask MA that allows it to pass through. The light passing through the image film 160 is focused by a lens 161, and interference fringes with the reference light described above are recorded on a hologram 162 placed on the focal plane of the lens 161. Next, in the same optical system, using an image different from the image originally used on the image film 160, the spot position of the light is determined by a detector 164 that two-dimensionally detects the light intensity distribution placed on the focal plane of the lens 160. The coordinates are determined by the controller 165. Such operations are repeated by changing the size and position of the mask on the image film 160, and the final measured value is recorded in the shift amount memory 166 as the shift amount.

The principle by which correlation calculation can be performed by a method using this optical system will be explained. When the monochromatic plane wave that has passed through the image f(r) is focused by a lens, the complex amplitude distribution or(q+) at the focal plane can be expressed by the equation 0.

1---■ However, A: amplitude of incident light, f: focal length of lens, which is exactly the same as two-dimensional Fourier transform. Therefore,
In FIG. 24, an image obtained by Fourier transforming the image f(1r) is formed on the focal plane of the lens 161, and the intensity distribution caused by interference with the reference light is recorded in the hologram.

Complex amplitude distribution of light waves on the hologram recording surface Hr (Ql)
is given by the following equation.

Hr (Ql) ocur (Ql)” A eXP
(-J (2Ql)--10 It is the angle formed with the Z axis.

Assuming that the photosensitive material of the hologram has an amplitude transmittance characteristic proportional to the 1/2 power of the light intensity, the amplitude transmittance distribution T□(ql) of this hologram is expressed by the equation 0.

To (IIIL)” l ut (ql) l 2+
A"+ATo (qOeXp(Jl'rQ1)+＾ut
I(ql)exp(-j aq+) ------9Next, transfer this hologram to another image g(r) on image film 1
Place it at 60 and light it. The wavefront 0 (Ql) of the light wave passing through the hologram at this time is expressed by the following equation.

0(q+)CC09・Tm(q+)=(lI
Jr (Ql) l"+A") ・L (Qj)+A
・Or (Ql)' 09(Ql)eXpD
J αqI)+＾口r”(ql)IJq(Ql )e
Xp(-J αql) ----■Here, if we place the lens 163 in the direction in which the wavefront of the diffracted light in the third term comes out and consider the two-dimensional Fourier transform effect caused by this, the complex amplitude distribution at the focal plane of the lens 163 is 0. ” (lr) is given by the formula 0.

0' (v)ccAf(1r)*g(1r-α)
−−−−−■ Therefore, if the origin is moved to α using the 0 formula, 0
The formula is 0' (f)ocAf(v) * g(h)
-----@, which is the same form as the correlation calculation shown in equation 0.

Therefore, a two-dimensional detector 164 is installed on this surface,
By detecting the point where the light intensity is maximum, f(Ir
) and g(Ir).

According to this optical correlation calculation, results can be obtained instantaneously.

Second Example FIG. 26 shows a block diagram of a second example of a correlation calculator using an optical system. In this example, instead of using the image film 160 for the target image in FIG.
more directly through the CRT controller 167 to the CRT 16
8 is read out to an incoherent-coherent converter 169 using a laser beam. This incoherent = coherent converter 1
69 uses a liquid crystal and is called CLV or BSG.

According to this configuration, a general operation such as once printing an image on a body film is not necessary, and a conversion from an electrical system to an optical system can be smoothly performed.

(4) Image Display FIG. 27 shows a block diagram of the image display device. The image display controller 170 determines which content to display from the manually inputted color image data, shift amount data, and θ value in response to instructions from the man machine interface 171, which includes a keyboard, joymatic, etc. The contents are displayed on the color display 172. There are three display modes displayed on the color display 172, as shown in FIGS. 28(A), (B), and (C). The first mode is the display of a raw color image as shown in FIG. 28 (8). The second mode is the 28th
As shown in FIG. 8, the unevenness information of the image obtained from the shift amount data and θ is displayed as a grayscale image. In other words, this is a mode in which the taller parts of the image are displayed brighter. The third mode is for displaying height information in three-dimensional graphics as shown in FIG. 28(C). The third mode has two more modes, one of which is to display a line drawing. At this time, the image display controller 170 performs smoothing processing and seed line processing on the three-dimensional image. In the second of the third modes, the three-dimensional image is displayed with the original color information overlaid, and the image display controller 170 adds smoothing processing and performs shading processing to create a more three-dimensional effect.

Note that in the third mode, three-dimensional graphics can be displayed from any angle direction according to instructions from the man machine interface 171.

Furthermore, in the second and third modes, by setting the cursor CA at an arbitrary position and instructing what to measure, values such as height, distance, area, etc. of the specified portion can be displayed.

On the other hand, the content displayed on the color display is recorded by the image recording device 173. Image recording devices include cameras that record on film, instant cameras,
Color hard copy etc. can be selected.

Note that the pattern on the pattern disk 19 in the first embodiment and the black and white pattern of the angle wire 17 and the sensor 21 in the fourth embodiment are not limited to these. A magnetoelectric conversion element may also be used. Further, instead of rotating the rotating drum 18 by hand using the angle knob 20, it may be driven by a motor or the like. Furthermore, in order to detect the amount of curvature, a strain gauge or the like may be provided in the curved portion 5 of the endoscope to detect the amount of deflection.

Furthermore, in the first to fourth embodiments, the endoscope curve B5 was curved at one location, but as shown in FIG. 29, the curve at the curved portion 5 was curved at two locations L! i1. W
2, and two types of endoscopic images with parallax may be obtained with the optical axes of the objective lens 1 being parallel.

An enlarged view of the distal end of the endoscope at this time is shown in FIG. In addition, in FIG. 30, an image guide fiber is not used, but a solid-state image sensor 2 is disposed at a position where an image is formed through an objective lens l at the tip of a curved portion 5 of an endoscope. This example uses a so-called electronic scope.

In this way, the optical axis of the objective lens 1 is parallel to the 2
When capturing different types of images, the shaded area that is a common field of view at two positions and the area in contact with the subject 4 are made common, and the image of the subject 4 has a parallax l of A-B. You can get it.

Further, in the above embodiments, the endoscope has a curved tip, but the present invention is not limited to this, and an endoscope with a curved tip may be used. Further, the direction of curvature or bending may be any of the up, down, left and right directions, or a combination of these directions.

Further, in the first embodiment, the amount of rotation of the rotary drum 18, that is, the amount of movement of the endoscope curved portion 5, is stored in the first and second latches 24.25 as the values of the counter 23, so that the amount of rotation of the rotary drum 18, that is, the amount of movement of the endoscope curved portion 5, is stored in the first and second latches 24, 25, and the values of the counter 23 are stored at two positions. The endoscopic images are stored in the first and second frame memories 14.
．． 15, the data is stored in separate storage means, but it is also possible to record all the data together in an optical disk device or a magnetic disk device, and later reproduce it and input it to the image processing device 16. It's okay.

Furthermore, it is also possible to calculate the difference between the values stored in the first and second latches in the first embodiment and convert it into curvature angle information. This may be performed on the image processing device 16 side, or may be performed on the endoscope device side. In addition, not only the angle information but also the direction of the optical axis of the objective lens 1 can be input by making the value of the counter 23 correspond to the curved position of the endoscope curved portion 5 on a one-to-l basis. As a result, more accurate calculations can be performed on the image processing device 16 side, taking into consideration the direction of the optical axis of the objective lens 1.

In addition, in the first to fourth embodiments, the endoscope image captured by the TV camera 12 or the solid-state image sensor 45 is converted into a digital signal by the A/D converter, but the endoscope image is converted into a digital signal by the A/D converter. The data may be recorded on a type optical disk device or an analog type magnetic disk device.

Further, in the above embodiment, the case where the state of the curved endoscope section 5 is two types is described, but the present invention is not limited to this, and the endoscopic images and the amount of movement of the curved endoscope section 5 in three or more types of states are described. If the image processing device calculates this in such a way as to obtain more accurate distance information and unevenness information, it is possible to obtain more accurate distance information and unevenness information.

〔Effect of the invention〕

As described above, according to the present invention, a plurality of images can be easily captured together with information relating to each other, and thereby various information regarding the subject can be obtained.

[Brief explanation of drawings]

FIG. 1 is an overall configuration diagram showing a first embodiment of the present invention, FIG. 2 is an enlarged view of the distal end of the endoscope in FIG. 1, and FIGS. FIG. 4 is an overall configuration diagram showing the third embodiment, FIG. 5 is an overall configuration diagram showing the fourth embodiment, and FIG. 6 is the amount of movement of the tip of the endoscope. 7 is a diagram showing the measurement method in the image processing device used in the first to fourth embodiments, FIG. 8 is a block diagram of the processing system centered on the image processing device, and FIG. 9 is a diagram showing the measurement method in the image processing device used in the first to fourth embodiments. Figure 10 is a block diagram of the distortion corrector in Figure 8, and Figure 1 shows image distortion correction.
Figure 1 shows the b-spline function, Figure 12 shows the 8th
Figure 13 is a block diagram showing the first example of the correlation calculator using an electric circuit, Figure 14 is a block diagram showing the second example, and Figure 15 is the same ( A block diagram showing the third example, FIG. 16 is also a block diagram showing the fourth example.
FIG. 17 is a block diagram showing an example of the spatial frequency filtering device shown in FIG. 16, FIG. 18 is a block diagram showing another example, and FIG. 19 is a block diagram showing another example. Block diagram: FIG. 20 is a block diagram showing a fifth example of a correlation calculator using an electric circuit; FIG. 21 is a block diagram showing an example of the spatial frequency filtering device shown in FIG. 20; FIG. 22 is another example. 23 is a block diagram showing a sixth example of a correlation calculator using an electric circuit, FIG. 24 is a block diagram showing a first example of a correlation calculator using an optical device, and FIG. 25 is a block diagram showing a first example of a correlation calculator using an optical device. FIG. 26 is a block diagram showing a second example of a correlation calculator using an optical device; FIG. 27 is a block diagram of the image display device shown in FIG. 8; FIG. A), (B) and (C) are diagrams each showing an example of image display in each mode, FIG. 29 is a diagram showing another example of the endoscope tip in the present invention, and FIG. FIG. 3 is an enlarged view of the distal end of the endoscope. Patent applicant Olympus Optical Industry Co., Ltd. Figure 2 Figure 6 Figure 7 Figure 8 Figure 9 Figure 1θ Figure 12 Figure 13 Figure 14 Figure 17 Figure 19 Figure 21 Figure 22 −−−−−＋＋＋＋−”Figure 24Figure 27Figure 28

Claims (2)

[Claims] (1) Capturing a first image through an endoscope at a first position, and capturing the first image at a second position while detecting the position relative to the first position through the endoscope. An endoscopic image capturing method characterized by capturing a second image overlappingly at least a portion of the image. (2) The endoscopic image capturing method according to claim 1, wherein the endoscope is moved from the first position to the second position by curving the distal end of the endoscope. .